PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual-Form ASR: Semantics-Aware Inverse Text Normalization for Chinese Speech Recognition

Fengrun Zhang, Li Fu, Wangjin Zhou, Lu Fan, Youzheng Wu, Xiaodong He

arXiv:2609.02901v1cs.CLcs.SD

TL;DR

ASR systems need both faithful spoken-form transcripts and readable written-form transcripts, but cascaded ITN exposes normalization to recognition errors and struggles with semantic numeric expressions. DF-ASR jointly trains paired transcript forms with prompt control, using LLM-generated supervision and ITN-MWER, and outperforms open-source ASR-ITN systems while preserving protected spans. Its demonstrated scope is offline Chinese ASR-ITN, with multilingual extension and residual supervision errors remaining limitations.

  • Problem

    Cascaded ASR-ITN makes written-form normalization vulnerable to recognition errors, while semantically dependent numeric expressions require context-sensitive decisions.

  • Method

    DF-ASR jointly trains a prompt-conditioned recognizer on paired spoken-form and written-form targets, using LLM-driven generate-and-judge supervision and ITN-MWER feedback.

  • Results

    DF-ASR achieves 4.64% I-CER, 94.85% keyword F1, and 95.18% forbidden-span preservation on the decision-aware evaluation protocol.

  • Takeaways & Limitations

    DF-ASR preserves prompt-level selection between spoken-form and written-form outputs while supporting semantics-aware normalization and protected-span preservation.

  • Takeaways & Limitations

    The work focuses on offline Chinese ASR-ITN; multilingual extension requires language-specific conventions and evaluation sets, and LLM-verified supervision may retain residual numeric errors.

Abstract

from arXiv · show

Modern automatic speech recognition (ASR) scenarios require both spoken-form transcripts for faithful transcription and readable written-form transcripts with inverse text normalization (ITN). However, these forms are typically produced by cascaded modules, where a spoken-form ASR output is rewritten by a separate ITN component, making written-form ASR-ITN vulnerable to recognition errors and decoupling normalization from acoustic-contextual modeling, especially for semantically dependent numeric expressions. In this paper, we propose Dual-Form ASR (DF-ASR), a framework that extends spoken-form ASR capability to semantics-aware written-form ITN through paired spoken-form and written-form supervision while retaining prompt-level selection between transcript forms. The dual-form supervision is constructed via a large language model (LLM)-driven generate-and-judge workflow, and training is further enhanced by ITN-MWER, a sequence-level objective that assigns higher cost to errors on normalization-sensitive spans. We also introduce a decision-aware REQUIRE-ITN/\FORBID-ITN protocol to separately measure required normalization and forbidden-span preservation. On manually annotated Chinese subsets from SpeechIO, DF-ASR consistently outperforms open-source ASR-ITN systems, remains competitive with strong closed-source references, and preserves reliable prompt-level control between spoken-form and written-form outputs.

I. INTRODUCTION

DF-ASR addresses the limits of cascaded Chinese ASR-ITN by jointly modeling faithful spoken-form transcription and semantics-aware written-form normalization under prompt control. It combines dual-form supervision, ITN-aware training, and separate evaluation of required normalization and forbidden-span preservation.

  • Cascaded ASR-ITN separates recognition from normalization, allowing recognition errors to propagate and preventing joint optimization of acoustic recognition and numeric-expression decisions.
  • Semantics determines whether the same spoken numeric expression should be normalized or preserved, making context-sensitive exceptions difficult for rigid rule-based ITN systems.For example, “ten to one” can become “10:1” for a ratio but remain spoken when meaning “very likely”.
  • DF-ASR jointly trains one prompt-conditioned recognizer with paired spoken-form and written-form targets, preserving faithful recognition while teaching written-form normalization policies.The prompt selects the transcript form, while semantic interpretation determines normalization within written-form mode.
  • Dual-form supervision is constructed with an LLM-driven generate-and-judge workflow, and ITN-MWER assigns higher cost to errors on normalization-sensitive spans.
  • 4.64% I-CER and 94.85% keyword F1 are achieved on REQUIRE-ITN, alongside 95.18% forbidden-span preservation on FORBID-ITN.

II. RELATED WORK

Prior ASR and ITN systems provide formatting or spoken-to-written conversion, but their control and optimization remain limited for context-dependent normalization. DF-ASR is motivated by explicit dual-form supervision and evaluation that distinguishes required rewriting from harmful over-normalization.

  • LLM-based ASR has expanded toward contextual, speaker-aware, code-switching, multilingual, and richer formatted transcription.
  • Whisper and open LLM-based ASR systems can learn or elicit formatting, but provide weak control over the boundary between spoken-form transcription and written-form rewriting.
  • Standalone ITN systems use rules, WFSTs, neural converters, or pretrained language models, yet operate after recognition and cannot jointly optimize normalization with acoustic recognition.
  • ASR-ITN supervision must constrain written targets to change appropriate numeric spans while preserving all non-ITN content.
  • Evaluation should separately measure required normalization and forbidden-span preservation because conventional metrics can miss harmful over-normalization.

III. METHOD

DF-ASR constructs paired spoken-form and written-form supervision through controlled LLM rewriting, verification, and deterministic filtering. Prompts explicitly select the transcript form.

  • DF-ASR models prompt-conditioned dual-form generation, using spoken-form targets for faithful recognition and written-form targets for semantics-aware ITN.
  • LLM-driven supervision: The workflow starts from fixed spoken-form transcripts and generates written-form counterparts intended to change only local ITN-related spans.
  • LLM-driven supervision: A text-only generator proposes candidates, while a pairwise verifier checks semantic preservation, appropriate normalization, and avoidance of nonlocal rewriting.
  • LLM-driven supervision: Candidates are retained when verifier scores exceed threshold τ and lightweight deterministic checks remove non-ITN rewriting artifacts.
  • Paired training: Each retained example yields spoken-form and written-form training instances, with the prompt specifying which transcript form to generate.

B. Prompt-conditioned ASR-ITN Model

The prompt-conditioned model combines speech and text representations in one architecture, allowing the same system to generate either transcript form. Dual-form cross-entropy makes the requested form explicit.

  • DF-ASR uses a speech encoder, adaptor, and LLM decoder, concatenating projected acoustic representations with textual prompt embeddings.
  • The same model generates spoken-form or written-form transcripts according to the prompt.
  • Dual-form cross-entropy prevents spoken and written formats from being treated as noisy alternatives under one unspecified condition.

C. ITN-MWER for Numeric Information Loss

ITN-MWER adds sequence-level feedback for numeric information, addressing the unequal semantic cost of numeric errors while preserving dual-form behavior through both prompt conditions.

  • Numeric errors such as altered decimal points, percentage signs, or units can change transmitted values rather than merely substitute characters.
  • ITN-MWER combines general transcription quality with numeric-keyword correctness for each spoken-form and written-form prompt condition.
  • Numeric-keyword reward extracts Arabic numbers and attached units or symbols, then scores matched, missing, and extra keywords.
  • ITN-MWER averages reward across an N-best set as a variance-reducing baseline and retains CE as a stabilizing term.
  • The reward uses CER-based fidelity alongside numeric-keyword feedback, with the latter emphasized for written-form normalization-sensitive spans.
  • The training illustration shows 4 candidates for readability, while actual training uses 16 candidates.

A. Training Data and Evaluation Set

DF-ASR is trained on a large Chinese speech corpus with paired transcript targets and evaluated on manually annotated subsets separating required normalization from forbidden-span preservation.

  • Training data: WENETSPEECH provides 14.61M utterances and 9992.6 hours of speech for DF-ASR training.
  • Training data: The final dual-form training set contains 14.23M utterances and 9574.7 hours, each paired with spoken-form and written-form targets.
  • Evaluation set: The manually annotated SpeechIO benchmark follows GB/T 15835–2011 and separates complementary ASR-ITN deployment requirements.
  • Evaluation set: REQUIRE-ITN contains 772 utterances requiring written-form normalization, while FORBID-ITN contains 309 utterances with 332 forbidden spans.
  • Systems compared: Evaluation compares DF-ASR with cascaded systems, open-source direct ASR systems, and closed-source models.

C. Implementation Details

The evaluation uses separate metrics for required normalization and protected-span preservation, alongside LLM-based generation and verification procedures. Training combines a frozen decoder with trainable speech components and evaluates ITN-sensitive regions separately from non-ITN regions.

  • Implementation Details: Gemini 3.0 Flash operationalizes GB/T 15835–2011 into prompt templates, while Qwen3.5-35B-A3B generates and verifies written-form candidates.The verifier scores semantic preservation and appropriate numeric-expression normalization on a 1–10 scale.
  • Implementation Details: DF-ASR initializes FireRedASR2 with a Qwen2-7B-Instruct decoder, freezes the decoder except for LoRA adapters, and trains the speech encoder and adaptor.The LoRA configuration uses rank 64, scaling factor 16, and dropout 0.
  • Implementation Details: ITN-MWER uses equal CER-based and numeric-keyword rewards with α = 0.5, β = 0.5, and CE stabilizing weight λ = 0.2.Vanilla MWER and keyword-only MWER are also evaluated by setting (α, β) to (1, 0) and (0, 1).
  • Implementation Details: REQUIRE-ITN reports CER, I-CER, NI-CER, and numeric keyword F1 to separate normalization-sensitive errors from other transcription errors.I-CER covers ITN-related regions, NI-CER covers non-ITN regions, and keyword F1 measures numeric values and units.
  • Implementation Details: FORBID-ITN reports CER and FSPR, where higher FSPR indicates better preservation of protected spoken-form spans.The protocol distinguishes incorrectly digit-normalized spans from spans deleted, substituted, or incompletely preserved.

V. RESULTS AND ANALYSIS

DF-ASR improves required numeric normalization while limiting changes outside ITN-related regions. It remains competitive with strong closed-source references, although DOUBAO-LLM ASR leads on overall CER and keyword F1.

  • Required Normalization: 4.64% I-CER, 2.35% overall CER, and 94.85% numeric keyword F1 show DF-ASR improving over the WFST cascade.The WFST cascade records 8.19% I-CER, 3.09% overall CER, and 89.71% numeric keyword F1.
  • Required Normalization: DF-ASR achieves lower I-CER and NI-CER than the LLM cascade, indicating more stable integrated ASR-ITN than post-processing recognized text.
  • Required Normalization: 1.86% NI-CER gives DF-ASR fewer non-ITN changes than WHISPER-LARGE-V3’s 5.07%, despite competitive keyword F1.Among closed-source models, DOUBAO-LLM ASR achieves the best overall CER and keyword F1 on REQUIRE-ITN, while DF-ASR remains competitive.

B. Forbidden-span Preservation

DF-ASR balances required numeric conversion with preservation of semantically protected spans. Control and ablation results indicate that dual-form supervision and reward balancing support this behavior without substantially degrading ordinary ASR.

  • Forbidden-span Preservation: 95.18% FSPR makes DF-ASR tied for the highest protected-span preservation score, above the LLM cascade’s 86.45%.DOUBAO-LLM ASR has lower CER at 3.56% but lower FSPR at 93.98%.
  • Required-vs-Forbidden Decision Analysis: FUNASR-NANO reaches 95.18% FSPR but only 54.63% keyword F1, whereas DF-ASR combines 95.18% FSPR with 94.85% keyword F1.The contrast shows the trade-off between preserving forbidden spans by avoiding ITN decisions and performing required numeric conversions.
  • Required-vs-Forbidden Decision Analysis: DF-ASR remains close to FireRedASR2 on the no-number control set, with CER of 3.07% versus 2.92%, and neither system inserts false digits.The result indicates no substantial degradation of ordinary ASR behavior under the written-form prompt.
  • Ablation Study: Verified dual-form targets improve I-CER and F1 over unverified LLM targets while slightly lowering FSPR, producing a better normalization-preservation balance.Unverified targets achieve the highest FSPR but retain an I-CER of 6.63%, reflecting conservative behavior that misses required normalization.
  • Ablation Study: ITN-MWER achieves the best I-CER and keyword F1 by combining general character-fidelity and numeric-keyword rewards.Vanilla MWER emphasizes character-level fidelity, whereas keyword-only MWER over-focuses on numeric tokens.

E. Qualitative Analysis

DF-ASR demonstrates context-aware dual-form transcription through examples where numeric expressions are selectively normalized or preserved. The examples also show compositional generalization and define the framework’s evaluation and scope boundaries.

  • DF-ASR normalizes a duration as “10年” but preserves “失去的十年,” showing that numeric-looking spans are handled according to context.
  • DF-ASR correctly generates the decimal, power expression, and percentage for unseen value “0.9999995,” demonstrating compositional generalization.
  • Experiments show DF-ASR outperforms open-source ASR-ITN baselines, remains competitive with closed-source references, and preserves protected spans under prompt control.
  • The study is limited to offline Chinese ASR-ITN, while multilingual extension requires language-specific conventions and evaluation sets, and supervision may retain numeric errors.
Loading 2609.02901v1…