PaperlayerView exact revision on arXiv

Source-linked AI summary

Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing

Laurin Wagner, Mario Zusag, Bernhard Thallinger

arXiv:2607.18934v1cs.CL

TL;DR

ASR systems often leave verbatim versus intended transcription policy implicit, destabilizing decoding and complicating evaluation and timing. This paper makes policy controllable with mode tags and timing supervision, reaching 94% zero-shot German disfluency F1 from 10%.

  • Problem

    ASR pipelines often leave verbatim-versus-intended transcription policy implicit, making disfluency representation and word-level timing inconsistent.

  • Method

    The paper uses paired-transcript decoder mode tags, supervised cross-attention for timing, and verbatimize for transcript-conditioned verbatim generation.

  • Results

    94% zero-shot German disfluency F1 is achieved from 10%, alongside precise word-level timing and scalable corpus enrichment.

  • Takeaways & Limitations

    The findings support treating transcription style as a controllable policy and using the resulting system to enrich speech corpora at scale.

  • Takeaways & Limitations

    Cross-lingual transfer was evaluated only on German, a language typologically close to English, and timing supervision mostly used MFA-generated alignments.

Abstract

from arXiv · show

Modern ASR models trained on heterogeneously annotated data treat transcription style (verbatim vs. intended) as an uncontrolled latent variable, causing measurable decoding instability, evaluation confounding (up to 60% of reported WER attributable to style mismatch), and unreliable word-level timing. We show that models already encode both styles; the challenge is controlled activation. Using coverage-aware decoder task tokens trained on parallel verbatim/intended transcript pairs, we raise German disfluency F1 from 10% to 79% zero-shot, despite English-only training. Full English-only fine-tuning surpasses all baselines in verbatim accuracy, disfluency detection, and intended-mode quality across both languages. We further introduce supervised cross-attention fine-tuning that improves word-level timestamps on disfluent speech beyond forced-alignment baselines. Finally, we propose verbatimize, a new task enabling scalable creation and enrichment of speech corpora with high-quality canonical verbatim transcriptions.

1. Introduction

The introduction frames transcription style as an implicit policy choice that mixed-style ASR training fails to control, causing unstable disfluency rendering and evaluation problems. It argues that pretrained models already contain verbatim capability and presents controllable mode tags, coverage-aware supervision, and corpus-enrichment tools as the solution.

  • Motivation: Transcription systems must choose between verbatim output that preserves disfluencies and intended output that retains fluent content.Verbatim transcripts include filled pauses, repetitions, self-corrections, cut-offs, and paralinguistic sounds, whereas intended transcripts prioritize readability and downstream text processing.
  • Problem: Mixed transcription conventions without a control signal make ASR systems inconsistently omit, transcribe, or hallucinate disfluencies.The introduction identifies heterogeneously annotated training data and scarce disfluency corpora, particularly beyond English, as central limitations.
  • Problem: 15% median character-level beam divergence occurs for the same disfluent audio, reflecting competing transcription strategies and style ambiguity.The introduction also notes that the same hesitation may be emitted, omitted, or rendered in non-canonical forms such as “um, uuumm, uhmm...”.
  • Result: German disfluency event F1 rises from 10% to 79% by training only mode-tag embeddings with encoder and decoder weights frozen.This result supports the claim that the core challenge is controlling an existing capability rather than acquiring new learned representations.
  • Contribution: Discrete decoder-prefix mode tags provide explicit switching between verbatim and intended output using paired supervision partitioned for annotation coverage.The introduction also describes extracting typed disfluency labels, disambiguating repetitions, and decomposing WER into content and style components for consistent evaluation.

2. Related Work

Prior work largely treats transcription style, timing, and disfluency labeling as fixed or externally handled, leaving gaps in bidirectional style control and integrated alignment. This work addresses these gaps with mode tags and supervised cross-attention without external components or tokenizer changes.

  • Transcription style: Most ASR systems use a single implicit transcription policy, while existing verbatim approaches generally lack bidirectional control between verbatim and intended modes.Prompting and soft prompts can elicit disfluencies but do not provide stable switching under paired supervision.
  • Word-level timing: Existing timestamp methods either extract unreliable emergent alignments or add language-specific forced aligners, increasing dependencies.Cross-attention alignment has been extracted with dynamic time warping, while WhisperX and Canary use separate forced aligners.
  • Contribution: Mode tags stabilize output token sequences, while supervised cross-attention makes emergent alignment trainable without external components or tokenizer changes.These mechanisms jointly target transcription-style control and word-level timing.
  • Disfluency modeling and evaluation: Prior disfluency systems commonly assume a fixed labeling policy and narrower event inventories than a unified set spanning repetitions, cutoffs, filled pauses, and paralinguistic events.End-to-end, multitask, and speech-generation approaches have incorporated disfluency or non-lexical sound labels.

3. The Transcription Policy Problem

Current ASR systems treat transcription policy as an uncontrolled source of decoding instability, evaluation error, and unreliable word-level timing. Explicit policy control stabilizes disfluency transcription, reduces beam divergence, and exposes substantial style-driven variation in reported WER.

  • Decoding instability: Without mode tags, 15–85% verbatim-training mixtures create unpredictable disfluency transcription, whereas mode tags maintain event F1 around 80% across mixing ratios.At 10% verbatim data, the untagged model achieves only 2% event F1, while German shows the same stability pattern despite no German verbatim training data.
  • Decoding instability: 46% reduction: verbatim mode tags lower Whisper beam divergence from 15.1% CER to 8.1% on disfluent AMI speech.Intended-mode conditioning lowers divergence to 11.2%, showing that explicit directives resolve ambiguity in both transcription directions.
  • Evaluation confounding: 3.7% WER on TED-LIUM and 12.1% on AMI separate verbatim and intended references of identical audio despite sharing 100% of content words.These reference disagreements are entirely stylistic, so standard WER conflates content errors with transcription-style differences.
  • Evaluation confounding: 60% of reported WER on AMI reflects style mismatch rather than information loss, while Whisper’s TED-LIUM WER shifts from 5.7% to 7.3% solely with reference choice.Whisper’s TED-LIUM content loss remains identical at 3.8% across those reference choices.
  • Word-level timing: 203 ms MAE on TIMIT and 568 ms on FluencyBank show that base Whisper’s cross-attention provides unreliable word-level boundaries, especially on disfluent speech.The paper attributes reliable timing to a stable output token sequence and explicit alignment supervision.

4. Method · 4.1. Activating style control: mode tags · 4.2. Vocabulary extension

The method activates latent ASR transcription styles with explicit decoder mode tags, coverage-aware tag partitioning, paired training, and an expanded vocabulary for disfluencies and sound events. It also supports transcript-conditioned verbatization and time-aligned output.

  • 4. Method: The method initializes from Whisper-medium and adds atomic vocabulary tokens for phenomena absent from the original tokenizer.It combines style control via mode tags, supervised cross-attention for timing, and transcript-conditioned disfluency recovery.
  • 4.2. Vocabulary extension: The vocabulary adds 2 filled-pause tokens and 12 paralinguistic sound-event tokens, including [uh], [um], [laughter], [cough], and [sneeze].The sound-event inventory also includes [sigh], [breath], [lipsmack], [sniff], [throat-clearing], [yawn], [noise], [crying], and [scream].
  • 4.1.1. Mode tag design: Decoder prefixes explicitly select verbatim or intended transcription policy: verbatim preserves audible disfluencies and sounds, whereas intended omits reparanda, interregna, and paralinguistic events.For example, verbatim output may retain fillers, laughter, repetitions, and cut-offs, while intended output is clean fluent text.
  • 4.1.2. Coverage-aware tag partitioning: Heterogeneous annotation makes a single verbatim tag set contradictory: incomplete labels can penalize both predicting and omitting the same sound event.The contradiction arises because corpora differ in whether they annotate disfluencies, injected sounds, or reliable labels.
  • 4.1.2. Coverage-aware tag partitioning: Coverage-aware partitioning assigns [verbatim 1..3] to disfluencies and [sound 1,2] to paralinguistic detection, activating only tags supported by each sample’s annotations.Disfluency-only, injected-sound, and fully annotated samples therefore receive different tag subsets.
  • 4.2.1. Paired training: Paired training randomly selects verbatim or intended mode for each sample and supervises the corresponding transcript, exposing the same audio to both policies.Encoder representations retain available words, disfluencies, and sounds, while mode tags determine which content is emitted.
  • 4.2.1. Paired training: The decoder prompt places mode tags before an optional intended-transcript hint and standard Whisper tokens, producing time-aligned transcription in the prompted style.The intended-transcript hint is optional and is used for verbatimize.

4.3. Activating timing: supervised cross-attention

Supervised training of selected cross-attention heads makes word-level timing systematically more accurate after transcription policy stabilization. The method aligns averaged attention to word-span targets and combines pause modeling, attention sharpening, word aggregation, and Viterbi decoding without architectural changes.

  • Activating timing: supervised cross-attention: Supervised cross-attention training enables systematic improvements in word-level timing accuracy once transcription policy and token outputs are stable.The base model’s cross-attention alignment is emergent and unreliable; directly training selected alignment heads addresses this limitation.
  • Activating timing: supervised cross-attention: The training target marks encoder frames inside each token’s corresponding word span, with frame shift ∆=0.02 s.Each decoder token is mapped to exactly one transcript word, whose interval defines a binary frame target.
  • Activating timing: supervised cross-attention: k=10 cross-attention heads are selected by their unsupervised alignment correlation on TIMIT and added greedily by marginal improvement.The selected heads form subset H for supervised training.
  • Activating timing: supervised cross-attention: Averaged attention is optimized with mean cosine distance because it enables complementary head specialization, unlike isolated per-head losses.Cosine distance compares distribution shape rather than magnitude.
  • Activating timing: supervised cross-attention: At inference, pause detection is decoupled from attention without architectural changes, while attention sharpening uses temperature scaling with τ=3.Sharpened token attention is aggregated to word-level emissions, then Viterbi decoding alternates virtual blank and word states to absorb pauses.

4.4. Activating transcript-conditioned generation: verbatimize · 4.5. Staged training

The paper introduces verbatimize to reconstruct canonical verbatim transcripts from audio and intended text by inserting acoustically grounded disfluencies. A two-stage training strategy activates this capability while preserving multilingual performance and reducing catastrophic forgetting.

  • 4.4. Activating transcript-conditioned generation: verbatimize: Veratimize addresses scarce, costly verbatim annotation by reconstructing canonical verbatim transcripts from audio and intended transcripts.It copies intended content while inserting acoustically grounded disfluencies at their temporal locations.
  • 4.4. Activating transcript-conditioned generation: verbatimize: The decoder prompt uses mode tags and delimiters around the intended text, including a [verbatim 1..5] prefix and <sot>/<eot> markers.These tokens condition transcript generation on the requested transcription policy.
  • 4.4. Activating transcript-conditioned generation: verbatimize: The model inserts disfluency markers, including [uh], [um], repetitions, cut-offs, and sound tags, at audio-detected positions.Valid training pairs require the intended transcript to form a subsequence of the verbatim transcript, making disfluencies strictly insertions.
  • 4.4. Activating transcript-conditioned generation: verbatimize: In standard verbatimize, the prompt contains the intended transcript and the target is the verbatim transcript.The section also defines verbatimize-copy for clean samples, alongside subsequence validation for standard training pairs.
  • 4.5. Staged training: Stage 1 freezes pretrained parameters and trains only newly added token embeddings, including mode tags, delimiters, and sound markers, for 1 epoch.This preserves Whisper’s pretrained multilingual capabilities and supports the latent-capability experiment reported in Section 7.1.
  • 4.5. Staged training: Stage 2 unfreezes the decoder and continues training with a reduced learning rate, using pretrained tag embeddings for stable initialization and mitigation of catastrophic forgetting.Mode-tag embeddings alone substantially activate verbatim transcription before encoder or decoder updates.

5. Data

The data pipeline combines paired verbatim/intended transcripts, diverse English and clean-speech corpora, canonical normalization, quality filtering, and targeted supervision for timestamps and rare-word evaluation.

  • Data requirements: Training requires paired transcripts for every audio sample: one verbatim reference and one intended reference.This pairing is the basis of the training procedure.
  • Datasets: The English verbatim data comprise ICSI (72 h), AMI (100 h), CORAAL (150 h), and NSC (2,000 h), with training restricted to a 40h high-quality NSC subset.LibriSpeech provides 960 h of clean read speech, and Common Voice supplies multilingual clean speech.
  • Transcript construction: GPT-4o converts verbatim annotations into intended transcripts by removing fillers, sound tags, fragments, repetitions, false starts, and readability-irrelevant formatting.For clean corpora, verbatim and intended transcripts are treated as identical because disfluencies are rare.
  • Quality and supervision: <3% WER filtering against Whisper-medium predictions removes mislabeled or degraded clean-corpus samples, while MFA supplies word-level timestamps for 500 hours.Transcripts are normalized with canonical tags for filled pauses, fragments or cutoffs, and temporally positioned sound events.
  • Verbatimize evaluation: 1,843 rare-word occurrences across 1,342 samples and 1,591 unique word types are manually verified for verbatimize evaluation.GPT-4o identifies rare or domain-specific words in ICSI transcripts, and verification confirms they do not appear elsewhere in training.

6. Experimental Setup

The experiments evaluate disfluency detection, word-level timing, and verbatimize on separate English and German benchmarks, using corpus-level metrics and established ASR and alignment baselines.

  • Benchmarks: Three capabilities are evaluated separately: disfluency detection, word-level timing, and verbatimize.Benchmarks include English and German disfluency datasets, read and disfluent speech timing datasets, and the ICSI rare-word evaluation set.
  • Evaluation metrics: Disfluency labels are extracted automatically by edit-distance alignment and deterministic surface-pattern rules.Rules identify cutoffs, fillers, sounds, repetitions, and other disfluency categories from aligned verbatim transcripts.
  • Evaluation metrics: Metrics are micro-averaged at corpus level after lowercasing text and removing punctuation.Transcription quality includes vWER and vCER against verbatim references and iWER against clean references, with iSR, iDR, and iIR decompositions.
  • Evaluation metrics: Timing evaluation reports mean absolute boundary error across aligned word pairs, while verbatimize evaluation reports content loss rate and rare-word recall.These metrics quantify timing offsets, unrecovered intended-reference words, and preservation of manually verified rare words.
  • Baselines: Comparisons include Whisper-medium, Canary-1B, Reverb, AssemblyAI Universal-3-Pro, and CrisperWhisper, with MFA and WhisperX additionally used for timing.Reverb is evaluated as an English-only baseline.

7. Results

The results show strong gains in controllable disfluency detection, supervised word-level timing, and verbatimize-based rare-word transcription. These gains hold across English and German, including disfluent speech and zero German verbatim training data.

  • Style control: Stage 1 raises event F1 from 12.0% to 53.0% on English and from 10.3% to 78.9% on German using only mode-tag embedding training.The encoder and decoder remain frozen, with zero weight updates.
  • Style control: Full decoder fine-tuning without German verbatim data further raises event F1 to 90.7% on English and 93.8% on German.The progression is 12%→53%→91% for English and 10%→79%→94% for German.
  • Word-level timing: Supervised attention reaches 102 ms MAE on FluencyBank, outperforming MFA at 142 ms and CrisperWhisper+s at 122 ms on disfluent speech.MFA performs best on read speech with 19 ms MAE on TIMIT but degrades on FluencyBank.
  • Verbatimize: Verbatimize training reduces CLR from 9.4% to 1.5% and raises RWR from 6.8% to 94.1% on the ICSI rare-word set.Casing perturbation further improves CLR to 1.3% and RWR to 96.1%.
  • Verbatimize: Residual rare-word errors are predominantly single-character ambiguities caused by cut-off transcriptions, such as “t- trr- transformer” versus “d- trtransformer.”Manual inspection identifies these cases as inherently ambiguous.

8. Discussion and Conclusion

The discussion argues that pretrained ASR models often possess verbatim and intended-transcription capabilities but need explicit controls to express them reliably. It also presents verbatimize as a way to address scarce disfluency data while identifying cross-lingual, naturalistic-evaluation, and alignment-supervision limitations.

  • Controllability: 10% to 79% German event F1 is achieved by training only 27 new token embeddings, without updating the 764M encoder/decoder parameters.This frozen-model result supports explicit control signals as an effective intervention for inconsistent pretrained models.
  • Data enrichment: 96.1% rare-word recall indicates that verbatimize can help break the cycle of scarce disfluency annotations and labeling errors propagated through ASR-derived corpora.The method decouples verbatim transcription from intended-transcript accuracy.
  • Controllability: 10% to 94% zero-shot German disfluency F1, and 79% with zero weight updates, support controllable mode-specific transcription in pretrained ASR.The discussion frames transcription style as an uncontrolled latent variable linked to decoding instability, evaluation confounding, and ill-defined timing.
  • Limitations and future work: Cross-lingual transfer was evaluated exclusively on German, a typologically close language to English, limiting conclusions about transfer to distant language pairs.The German evaluation set was author-recorded, and most timing supervision uses MFA-generated rather than hand-annotated boundaries.

9. Generative AI Use Disclosure

The authors disclose limited generative AI use for coding assistance and formatting tables and plots, while stating that experiments and results were manually handled and the paper was manually written.

  • Generative AI Use Disclosure: Claude assisted with some experimental code, while tables and plots used generative AI formatting; experiments, result verification, and paper writing were manual.The authors assume full responsibility and accountability for the submission.
Loading 2607.18934v1…