PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluation

Zixuan Jiang, Yanqiao Zhu, Peng Wang, Qinyuan Chen, Xinjian Zhao, Xipeng Qiu, Wupeng Wang, Zhifu Gao, Xiangang Li, Kai Yu, Xie Chen

arXiv:2605.29430v1cs.AIcs.CL

TL;DR

Most ASR systems remain single-pass, limiting iterative resolution of meaning-critical errors and exposing weaknesses in token-level evaluation. This paper introduces Interactive ASR, Agentic ASR, S2ER, and an Interactive Simulation System; iterative interaction consistently reduces semantic errors across benchmarks, especially in early rounds.

  • Problem

    Most ASR systems use single-pass transcription, while WER assigns uniform costs and fails to reflect the unequal semantic impact of recognition errors.

  • Method

    The paper formulates Interactive ASR as multi-turn refinement and proposes Agentic ASR, S2ER, and an Interactive Simulation System for semantic correction and evaluation.

  • Results

    Iterative interaction consistently reduces semantic errors across multilingual, named-entity-intensive, and code-switching benchmarks, with most gains emerging in the first few rounds.

  • Takeaways & Limitations

    S2ER captures the primarily semantic gains of iterative interaction more faithfully than conventional lexical metrics.

Abstract

from arXiv · show

Automatic speech recognition (ASR) is a core component of human--computer interaction and an increasingly important front-end for LLM-based assistants and agents. However, most current ASR systems still follow a single-pass paradigm, which is poorly aligned with human communication, where misunderstandings are resolved through iterative clarification and refinement. This mismatch makes it difficult to correct meaning-critical errors once they occur. Meanwhile, token-level metrics such as WER or CER cannot adequately reflect such a problem. To address these limitations, we formulate \emph{Interactive ASR} as a multi-turn refinement task and propose \textbf{Agentic ASR}, a closed-loop framework that combines a single-pass ASR front-end with semantic correction, intent routing, and reasoning-based editing. We further introduce the \textbf{Sentence-level Semantic Error Rate} ($S^2ER$), an LLM-based semantic evaluation metric, together with an \textbf{Interactive Simulation System} for scalable and reproducible benchmarking. Experiments on multilingual, named-entity-intensive, and code-switching benchmarks show that iterative interaction consistently reduces semantic errors, with much larger gains in $S^2ER$ than in conventional token-level metrics. Human--AI alignment and ablation studies further validate the reliability of the semantic judge and the robustness of the proposed framework. The code is available at: https://interactiveasr.github.io/ and the live demo is available at https://i-asr.sjtuxlance.com/

I. INTRODUCTION · II. RELATED WORK · A. ASR Metrics

The paper reframes ASR as interactive, feedback-driven refinement because single-pass systems poorly resolve meaning-critical errors and token-level metrics inadequately capture semantic impact. It proposes Agentic ASR and S2ER, while positioning S2ER against prior semantic-sensitive evaluation methods.

  • I. INTRODUCTION: Modern ASR systems remain largely single-pass, unlike human communication, where misunderstandings are resolved through feedback and repair.This mismatch is especially problematic for meaning-critical recognition errors.
  • A. ASR Metrics: WER and CER measure token-level mismatch but do not distinguish minor surface deviations from errors that change intended meaning.Prior semantic evaluation studies likewise show that semantic preservation is not always captured by conventional token-level measures.
  • I. INTRODUCTION: Interactive ASR formalizes transcription as stateful multi-turn refinement under user feedback rather than independent single-pass decoding.The task formulation targets progressive hypothesis refinement.
  • I. INTRODUCTION: Agentic ASR is a closed-loop framework that combines semantic correction, intent routing, and reasoning-based editing to repair meaning-critical recognition errors.Its paradigm is designed to support progressive correction during interaction.
  • I. INTRODUCTION: The paper also establishes S2ER and ISS as an evaluation system and verifies its reliability and effectiveness through human–AI alignment and multi-benchmark experiments.This contribution complements the interactive task formulation and Agentic ASR framework.
  • A. ASR Metrics: WER assigns uniform costs to tokens and edit operations, failing to reflect unequal semantic impact across recognition errors.This limitation motivates semantic-sensitive extensions of edit-distance metrics.
  • A. ASR Metrics: Prior ASR metrics add semantic sensitivity through context-dependent weights, emphasis on named entities, semantic-distance signals, contextual similarity, embeddings, LLM grading, or downstream QA behavior.Examples include WWER, NE-WER, Semantic WER, H eval, BERTScore, SemDist, LASER, and Answer Error Rate.
  • A. ASR Metrics: S2ER uses a binary functional criterion: whether the hypothesis preserves enough meaning for correct intent execution in an interactive setting.Unlike prior approaches, it targets interaction success rather than fine-grained lexical similarity or constructed downstream proxy scores.

B. Human-Feedback-Based ASR Approaches … A. Task Formulation

The paper frames Interactive ASR as stateful, multi-turn refinement that incorporates user feedback to repair meaning-critical errors, and develops an intent-preserving semantic judge for evaluating such interaction. It builds on prior correction interfaces, feedback-based adaptation, iterative agents, and LLM-based evaluation while adding open-form repair and stability practices.

  • B. Human-Feedback-Based ASR Approaches: Prior human-feedback systems support N-best selection, touch-based text correction, and acoustic respeaking to address ASR errors.These approaches rely mainly on explicit correction interfaces or user re-utterance.
  • B. Human-Feedback-Based ASR Approaches: The Gift of Feedback uses on-device corrections through federated learning for long-tail model adaptation rather than online guidance during ongoing interaction.This distinguishes feedback used as training supervision from feedback used to resolve errors in real time.
  • III. AGENTIC ASR: Building on iterative reasoning agents, the framework accepts open-form natural-language feedback to repair ASR errors in place.This bridges rigid correction interfaces and language-driven interactive refinement.
  • C. LLM as a Judge: LLM-based ASR semantic evaluation can align better with downstream task success than purely lexical metrics, though hidden-state scoring is less interpretable.Prior work evaluates transcription quality using semantic representations, while LAT-TEScore frames assessment as binary semantic preservation.
  • C. LLM as a Judge: S2ER extends binary semantic-preservation evaluation by targeting intent-preserving usability for Interactive ASR.Its design follows the binary-evaluation principle while focusing on applicability to interactive transcription.
  • C. LLM as a Judge: The evaluation protocol improves LLM-judge reliability through rubrics, examples, order swapping, and multi-round aggregation.These practices address variance and positional sensitivity documented in prior evaluation studies.
  • A. Task Formulation: Conventional ASR determines each output from the current acoustic signal and does not explicitly incorporate later user feedback into the same decoding process.This single-pass paradigm treats speech inputs independently rather than as an ongoing interaction.
  • A. Task Formulation: Interactive ASR reformulates transcription as a stateful multi-turn refinement process conditioned on new speech and interaction history.Unlike conventional single-pass ASR, each turn explicitly uses historical context for progressive repair of meaning-critical errors.

B. Agentic ASR Framework

Agentic ASR combines a single-pass ASR front-end with an LLM module that iteratively updates transcription states using history. Its refinement pipeline performs semantic correction, intent routing, and structured reasoning-based correction.

  • Architecture: Agentic ASR combines a single-pass ASR front-end with an LLM reasoning-and-editing module to update transcription state from speech and transcription history.At turn t, the framework takes speech I_t and prior transcription history Y[:t−1] as input and outputs updated state Y_t.
  • Intent routing: Intent routing classifies the corrected instruction as confirmation, new input, or correction, determining whether to accept, append, or edit the transcription state.Confirmation accepts the current state; new input treats the utterance as newly transcribed content; correction edits the existing state.
  • Semantic correction: Semantic correction rewrites the raw ASR hypothesis into an explicit, executable instruction consistent with the interaction context.This step addresses semantic inconsistencies introduced when correction utterances are themselves recognized by the ASR front-end.
  • Reasoning-based correction: For correction intents, a Locate–Reason–Modify pipeline identifies the editable span, infers the intended change, and applies it to the transcription state.The explicit decomposition improves controllability and aligns correction with users’ partial repair instructions rather than an unconstrained one-step rewrite.

IV. SENTENCE-LEVEL SEMANTIC ERROR RATE (S2ER) · A. Definition

S2ER evaluates whether ASR transcriptions preserve intended meaning at the utterance level, rather than merely matching tokens. Its binary semantic judge emphasizes task-critical content and uses bidirectional multi-round voting to improve label stability.

  • A. Definition: S2ER measures meaning preservation at the utterance level.It is introduced as the Sentence-level Semantic Error Rate.
  • A. Definition: S2ER labels each ASR hypothesis as semantically equivalent or meaning-critical error.The binary label is 1 for semantic equivalence and 0 for a meaning-critical error.
  • A. Definition: S2ER reports the proportion of utterances whose transcriptions fail to preserve intended meaning.This definition targets sentence-level semantic failure rather than token-level mismatch.
  • A. Definition: The task-oriented judge checks main intent and key meaning-bearing content, especially proper nouns and named entities.It ignores disfluencies, filler words, punctuation, and other non-semantic variations.
  • A. Definition: S2ER uses a concise prompt specification because its target is binary semantic equivalence.The criterion prioritizes task-critical information instead of an exhaustive rule list.
  • A. Definition: The evaluation uses three-round bidirectional voting to improve robustness and reduce order sensitivity.Each round queries the judge twice with reversed input order.
  • A. Definition: A round counts as positive only when both reversed-order decisions indicate equivalence, and the final label uses majority voting across three rounds.This protocol mitigates input-order bias and improves label stability.
  • A. Definition: S2ER can distinguish preserved meaning from semantic failure when token-level metrics are misleading.Filler or discourse-word mismatches can produce high WER with preserved meaning, while one key-entity substitution can produce lower WER but semantic failure.

B. S2ER versus token-level metrics · C. Interactive Simulation System

S²ER evaluates whether transcriptions preserve meaning, intent, and key entities more directly than WER or CER. The Interactive Simulation System enables scalable, repeatable multi-round evaluation using semantic stopping and simulated corrective feedback.

  • B. S2ER versus token-level metrics: WER and CER measure surface-form mismatch but cannot distinguish negligible errors from meaning-critical errors affecting intent or key entities.This limitation is especially important for interactive and agent-oriented ASR, where downstream success depends on preserved intent and entities.
  • B. S2ER versus token-level metrics: 42.9% WER can still correspond to semantic correctness when mismatches affect only discourse or filler words.In Case A, “Let’s open the window?” preserves the reference intent, yielding ẑ = 1 and no S²ER error.
  • B. S2ER versus token-level metrics: S²ER captures functional usability more directly because many non-essential token errors may preserve meaning, while one local substitution can corrupt a key entity or intent-bearing word.This distinction explains why interaction can produce larger S²ER gains than WER, CER, or MER.
  • C. Interactive Simulation System: ISS provides scalable and repeatable benchmarking by simulating multi-round user–system interaction and applying the S²ER Judger as the semantic stopping criterion.It addresses the expense and reproducibility challenges of large-scale human-in-the-loop evaluation.
  • C. Interactive Simulation System: At each ISS round, the evaluated system produces a transcription that the S²ER Judger checks for semantic equivalence with the ground truth.If equivalence is absent, the User Simulator generates the next corrective spoken instruction.
  • C. Interactive Simulation System: ISS stops when semantic equivalence is achieved or when a predefined maximum number of rounds is reached.Each sample begins from initial user speech X_i,0 and ground-truth transcription Y_i,GT.

1) S2ER Judger in ISS: · 2) User Simulator:

The Interactive Simulation System uses an S2ER Judger to determine semantic equivalence at each round and terminate successful interactions, while a User Simulator generates corrective speech for continued refinement. The framework also supports round-wise semantic-error measurement and extension to token-level metrics.

  • 1) S2ER Judger in ISS:: At round t, the S2ER Judger assigns a binary semantic-equivalence label to the current transcription Yi,t.It applies the same protocol to (Yi,t, Yi,GT).
  • 1) S2ER Judger in ISS:: The label ˆzi,t = 1 denotes semantic equivalence between the current transcription and ground truth.
  • 1) S2ER Judger in ISS:: When ˆzi,t = 1, the interaction terminates and all remaining rounds are marked successful.
  • 1) S2ER Judger in ISS:: Otherwise, Yi,t is forwarded to the User Simulator to generate the next-round user input within a predefined maximum of T rounds.
  • 2) User Simulator:: The User Simulator combines an LLM-based corrector with a TTS vocalizer to produce simulated user speech Xi,t+1.
  • 2) User Simulator:: The corrector compares Yi,t with Yi,GT, identifies the key semantic discrepancy, and generates a concise correction instruction.
  • 2) User Simulator:: The TTS vocalizer converts the correction instruction into speech and feeds it back to the evaluated Interactive ASR system.
  • 2) User Simulator:: Round-wise S2ER measures the proportion of N dataset samples that remain semantically incorrect after round t.

V. EXPERIMENTS … B. Human–AI Alignment Study

The experiments evaluate Agentic ASR and S2ER across multilingual, named-entity-intensive, and code-switching speech, while testing whether the LLM judge aligns with human semantic judgments. The setup uses fixed ASR, LLM, and speech-synthesis components and a 120-example human–AI validation study.

  • V. EXPERIMENTS: The experiments assess both the Agentic ASR framework and the S2ER semantic evaluation protocol.The study also analyzes performance trends and robustness to ASR backbone, LLM scale, and judge strategy.
  • 1) Model Configuration:: Unless otherwise specified, Qwen3-ASR-1.7B generates initial hypotheses, while Qwen3-32B powers reasoning, correction, and semantic judging.Index-TTS-1.5 synthesizes speech using each sample’s reference audio as an acoustic prompt for speaker consistency.
  • 2) Evaluation Datasets:: Evaluation splits cover multilingual, named-entity-intensive, and code-switching conditions to test robustness and generalization.These three categories use representative benchmark datasets.
  • 2) Evaluation Datasets:: Multilingual evaluation uses GigaSpeech Test for English and WenetSpeech Test Net for Mandarin open-domain ASR.
  • 2) Evaluation Datasets:: Named-entity evaluation uses AISHELL-NER Dev† and AISHELL-NER Test†, filtered to utterances containing named entities.The splits are derived from AISHELL-1 development and test data with AISHELL-NER annotations.
  • 2) Evaluation Datasets:: Code-switching evaluation uses ASRU2019 Test and CS-Dialogue Test† for Mandarin–English speech.CS-Dialogue Test† selects code-switching utterances from the original CS-Dialogue corpus.
  • B. Human–AI Alignment Study: The human–AI alignment study samples 40 utterances each from GigaSpeech, WenetSpeech, and ASRU2019, yielding 120 validation examples.The set covers English, Chinese, and code-switching conditions.
  • B. Human–AI Alignment Study: Twenty-five non-experts and five domain experts independently annotate semantic consistency with binary labels, whose average becomes the human reference score.The LLM Judger is evaluated on the same set using Pearson correlations over five runs; its correlation exceeds 0.8 across datasets and is slightly above experts’ correlations, with small standard deviations.

C. Main Results

Agentic ASR consistently improves transcription through multi-turn interaction across multilingual, named-entity-intensive, and code-switching benchmarks. The largest and fastest gains occur in semantic correctness, with S2ER improving more sharply than token-level metrics.

  • Overall trends: Multi-turn interaction consistently improves transcription quality across multilingual, named-entity-intensive, and code-switching benchmarks, with the largest gains at the semantic level.Increasing the interaction budget consistently improves semantic correctness across the reported benchmark categories.
  • Semantic improvements: S2ER on GigaSpeech Test falls from 21.47% at Loop 0 to 12.35% after one interaction and 3.49% by Loop 10.S2ER decreases monotonically with additional interaction, with a large one-step gain and continued improvement through later loops.
  • Semantic improvements: S2ER on ASRU2019 Test falls from 28.57% at Loop 0 to 10.32% after one loop and 1.36% by Loop 10.The trajectory continues improving without reversal as interaction loops increase.
  • Interaction efficiency: Most semantic benefit arrives in the first few interaction rounds, followed by diminishing but still positive returns.This suggests that the framework recovers most semantic errors with a small interaction budget rather than requiring many rounds.
  • Challenging benchmarks: Named-entity-intensive subsets reach final S2ER values around 2%, while code-switching improvements remain substantial but are smaller for CS-Dialogue Test† than ASRU2019 Test.The results indicate effectiveness on proper-noun and other high-value content errors, with spontaneous conversational code-switching remaining more difficult.
  • Metric comparison: S2ER drops much more sharply than WER, CER, or MER, showing that interaction primarily repairs meaning-critical errors rather than merely polishing local token mismatches.This discrepancy supports evaluating interactive ASR with a semantic metric instead of token-level measures alone.

D. Ablation Study · 1) Different Base ASR Model: · 2) Size of LLM Reasoner:

Agentic ASR’s interaction gains persist across substantially different ASR backbones and with an 8B reasoner, showing that interaction itself drives semantic improvement. However, stronger base recognizers and larger reasoners achieve lower final semantic error floors and more reliable, precise corrections.

  • 1) Different Base ASR Model:: Agentic ASR remains effective with strong, moderate, and weak ASR backbones, as S2ER decreases steadily with interaction across all three benchmarks.The default Qwen3ASR-1.7B was replaced by FireRedASR2-LLM-8.3B and Whisper while all other pipeline components remained unchanged.
  • 1) Different Base ASR Model:: Whisper still achieves large, practically meaningful improvements despite starting with Loop-0 S2ER values of 47.77% on AISHELL-NER Test† and 46.32% on ASRU2019 Test.These results show that interaction can substantially improve weak transcriptions rather than merely polish strong hypotheses.
  • 1) Different Base ASR Model:: 0.55%, 2.02%, and 6.82% are the final AISHELL-NER Test† S2ER values for FireRedASR2-LLM-8.3B, Qwen3ASR-1.7B, and Whisper, respectively.Stronger backbones generally retain an advantage after the full interaction budget, although weaker models still benefit substantially.
  • 2) Size of LLM Reasoner:: The 8B reasoner preserves a monotonic S2ER decrease on all benchmarks, demonstrating that a compact model can support meaningful multi-turn correction and user-feedback simulation.Both the Agentic ASR Reasoning LLM and the User Simulator Correction Generator were replaced with Qwen3-8B.
  • 2) Size of LLM Reasoner:: At Loop 10, Qwen3-8B is worse than Qwen3-32B on every dataset, with absolute S2ER gaps ranging from 2.11 to 4.07 points.The penalty appears across multilingual, named-entity-intensive, and code-switching settings.
  • 2) Size of LLM Reasoner:: WER on GigaSpeech Test and MER on CS-Dialogue Test† increase as interaction proceeds under Qwen3-8B, despite continued S2ER improvement.This divergence indicates that smaller-model corrections can preserve global meaning while producing less precise local edits.
  • 2) Size of LLM Reasoner:: Weaker instruction following and edit precision make Qwen3-8B more likely to generate ambiguous instructions, misidentify target spans, or rewrite beyond the intended scope.These behaviors explain why interaction can repair sentence-level semantics while introducing local token mismatches and degrading WER, CER, or MER on some datasets.
  • 2) Size of LLM Reasoner:: Qwen3-8B preserves the main interaction advantage, but stronger LLMs provide more reliable correction and a lower final semantic error floor.LLM capability affects how cleanly and how far the correction process proceeds, rather than determining whether Agentic ASR works at all.

3) LLM-as-a-Judge Strategy: · VI. CONCLUSION

The S2ER judge benefits from limited repeated voting, with majority-3 improving validation correlation while additional rounds show diminishing returns. The paper concludes that Agentic ASR enables semantic refinement and identifies interactive supervision and smaller refinement models as future directions.

  • 3) LLM-as-a-Judge Strategy:: Majority-3 improves full-validation correlation from 0.8543 to 0.8628 over a single bidirectional judgment.The repeated-voting ablation tests whether additional rounds improve agreement with human judgments.
  • 3) LLM-as-a-Judge Strategy:: Majority-5 performs best on GigaSpeech, but majority-3 achieves higher full-validation correlation than majority-5 and majority-7.Additional voting rounds add cost more reliably than quality, making majority-3 the practical robustness-efficiency default.
  • VI. CONCLUSION: Interactive ASR is formulated as multi-turn semantic refinement through Agentic ASR, combining single-pass ASR, semantic correction, intent routing, and reasoning-based editing.The framework also introduces S2ER and an Interactive Simulation System for scalable, reproducible multi-turn evaluation.
  • VI. CONCLUSION: Iterative interaction consistently reduces semantic errors on multilingual, named-entity-intensive, and code-switching benchmarks, with most benefits emerging in the first few rounds.S2ER captures these primarily semantic gains more faithfully than WER, CER, NER, and MER.
  • VI. CONCLUSION: Ablations indicate that smaller LLMs remain usable, while stronger reasoners provide more stable editing.The supplied conclusion passage contrasts usability of smaller models with the stability and precision advantages of larger models.
  • VI. CONCLUSION: Future work should incorporate richer interactive supervision and post-train smaller task-specific refinement models.These directions target realistic deployment robustness and the stability and precision gap between compact and larger models.
Loading 2605.29430v1…