PaperlayerView exact revision on arXiv

Source-linked AI summary

Voice Memory for Agentic Speech Recognition

Chao-Han Huck Yang, Zih-Ching Chen, Piotr Zelasko, Zhehuai Chen, Jagadeesh Balam, Boris Ginsburg

arXiv:2607.26410v1cs.CLcs.AIcs.SDeess.AS

TL;DR

Speech recognizers still make systematic, domain-specific errors, while existing adaptation methods do not simultaneously provide inference-only, auditable, portable correction. Voice Memory externalizes an act-or-abstain policy into an asynchronously optimized text memory, and across nine domains its correction gains track recoverable headroom (r=+0.90).

  • Problem

    Existing adaptation methods do not provide correction that is simultaneously inference-only, auditable, and portable across correctors, despite systematic domain-specific recognition errors.

  • Method

    Voice Memory stores correction skill in a per-domain text memory that a frozen corrector reads while a score-gated optimizer improves it through bounded edits.

  • Results

    Across nine domains, correction gains track recoverable headroom (r=+0.90).

  • Takeaways & Limitations

    The learned correction policy is an auditable text artifact that can transfer across corrector families without changing model weights.

  • Takeaways & Limitations

    The main analyses use one open-source corrector, while recoverable-headroom measures depend on the sentence encoder used.

Abstract

from arXiv · show

We present Voice Memory, a inference-only scheme for agentic speech recognition: at stream time, a frozen corrector reads a single per-domain memory.md and decides per utterance whether to act on the hypothesis or abstain and keep the 1-best. Asynchronously, a score-gated optimizer revises that file through bounded edits, accepting an edit only when it strictly improves a held-out score. Extended from classical ASR-LM framework, we refer this split the listener-thinker architecture; the two roles are coupled only through the memory, so no weights change and the learned skill stays auditable and portable. Restraint turns out to be the operative skill this loop discovers: unconstrained generative error correction (GER) over-corrects, breaking correct tokens on up to 64% of its edits on financial news, and Voice Memory, reduces this rate to 35%. Across ten HyPoradise domains with an open corrector, Voice Memory, lowers weighted word error rate from 8.36% to 7.52% (7.47% with three added in-context examples) without regressing any dataset below its 1-best baseline; gains concentrate where recoverable headroom is largest, including air-travel commands (8.40% to 3.40%) and noisy far-field speech (CHiME-4, 12.69% to 10.46%). The memory transfers across corrector families and adds zero parameters to the inference path. A demo and example code are provided for future studies.

1. Introduction

The introduction frames agentic speech recognition as a listener-thinker system in which a frozen corrector chooses whether to act or abstain using persistent external memory. Voice Memory makes this adaptation inference-only, auditable, portable, and self-improving through score-gated bounded edits that favor restraint.

  • Motivation: In low-error speech recognition, generative hypothesis rewriting can over-correct tokens that were already correct, motivating selective action rather than a fixed transformation.Examples include changing “home” to “home’s” and normalizing digits or spelling.
  • Formulation: The listener-thinker format separates synchronous decoding and per-utterance act/abstain decisions from asynchronous revision of persistent text state.The listener and thinker are coupled only through the persistent artifact, making the system auditable and portable.
  • Method: Voice Memory stores correction skill in a per-domain text memory that a frozen corrector reads at test time, while an optimizer accepts only strictly improving bounded edits.The optimizer performs bounded add/delete/replace edits from scored rollouts and does not update model weights.
  • Method: Restraint is the dominant policy discovered from scored mistakes, expressed through suppressive rules that preserve verbatim ASR tokens when correction is harmful.The loop learns when the corrector should act and when it should hold still.
  • Contributions: Sub-10 KB is the reported size of the auditable artifact, alongside no training infrastructure and a low-cost adaptation path for ASR–LLM cascades.Voice Memory uses forward passes only on the frozen deployment endpoint and remains energy efficient on device.

2. Related Work

Related work spans agentic ASR, contextual biasing, robust hypothesis correction, text-space optimization, and critiques of WER. Voice Memory is positioned as differing from weight-based per-utterance agency and as applying score-optimized natural-language memory to ASR correction.

  • ASR-LM and Agentic Setup for ASR: Speech-Hands and Speech-Mind learn per-utterance action tokens by fine-tuning omni-models, making agency acoustic, weight-based, and re-decided each utterance.Voice Memory differs along these axes, according to the supplied passage.
  • Generative error correction and contextual biasing: Contextual biasing progressed from shallow-fusion and on-the-fly rescoring of domain n-grams to end-to-end model adaptations such as CLAS, RNN-T, and trie-based deep biasing.The passage characterizes these methods as adapting the acoustic model.
  • Noise-robust ASR and robust hypotheses: Robust-ASR correction addresses degraded n-best lists under acoustic noise, requiring denoising rather than simple reranking.Robust HyPoradise pairs noisy Whisper hypotheses with clean references across four named datasets, while RobustGER uses language-space noise embeddings.
  • Text-space optimization: Text-space optimization improves natural-language artifacts such as prompts and skills against a score instead of using gradient updates.For ASR correction, the n-best oracle sets a bounded target ρ, and restraint is the main learned behavior.
  • WER critique and semantics: Prior work questions WER as ASR’s sole metric, motivating analyses of surface-error–meaning gaps and semantic training signals.The supplied passage states that semantic training yields lower WER than a traditional WER signal.

3. The Voice Memory Method

Voice Memory frames speech correction as a per-utterance choice to act on a frozen model’s hypothesis or abstain, while storing learned behavior in an external Markdown memory. An inference-only feedback loop improves this memory through bounded, score-gated edits without changing model weights or adding inference calls.

  • 3.1 Decision rule: Correction is a per-utterance decision to act or abstain from changing the 1-best hypothesis.The corrector reads n-best hypotheses, but at low error rates its main decision is whether to change h1 at all.
  • 3.1 Decision rule: The Recoverable Information Ratio (ρ) measures the fraction of the 1-best-to-oracle WER gap that correction closes.ρ<0 denotes damage, ρ=1 exactly closes the gap, and ρ>1 goes below the n-best oracle bound by recovering unseen tokens.
  • 3.2 Memory optimization: A frozen corrector reads an optional short Markdown memory that remains external, editable, reusable across correctors, and absent from model weights.With s=∅, the method reduces to standard GER.
  • 3.2 Memory optimization: The optimizer revises memory through bounded edits and accepts a candidate only when its held-out selection score strictly improves.Training supplies rollout evidence, selection gates acceptance, and test is used once for reporting.
  • 3.3 Over-correction analysis: The Harmful Edit Rate (her) is the fraction of token edits that break already-correct tokens, isolating over-correction.Edits are labeled helpful, harmful, or missed by aligning the corrected transcript with the 1-best hypothesis and reference.

4. Over-Correction and Scaling Properties

Voice Memory addresses unconstrained correction’s over-correction by reducing harmful edits and avoiding degradation on low-headroom domains. Its gains scale with recoverable WER headroom, reaching their largest values where 1-best error is highest.

  • Over-correction and restraint: 0.64 harmful-edit rate occurs for zero-shot GER on wsj, while Voice Memory lowers the harmful-edit rate to 0.35 on wsj.GER also raises WER on ls_clean from 1.8 to 2.4, ls_other from 3.7 to 4.1, and td3 from 4.1 to 4.4.
  • Scaling with WER: +0.90 correlation links recoverable headroom 𝜌 with 1-best WER across nine Table 3 domains.Voice Memory recovers little at the ls_clean floor (𝜌=−0.11) and td3 (𝜌=−0.09), but recovers most of the gap on high-WER atis (𝜌=0.96).
  • Optimization measurement: 2WER and her use the efficient open-source agwer implementation, which is safe as a reward target inside the optimization loop.The package is released under the MIT license.
  • Robust scaling: Voice Memory lowers WER on CHiME-4 and NOIZEUS, where static GER over-corrects, but is near-null on VoiceBank-DEMAND and LS-FreeSound.The table reports 𝜌vm as the recovered fraction of the 1-best-to-oracle gap.

5. Memory Transfer and Portability

Voice Memory is portable across corrector families and extends beyond ASR to speech-translation correction. Its benefits depend on recoverable headroom and memory quality, with learned restraint outperforming broader or larger alternatives.

  • Cross-corrector transfer: 6.68% to 6.08% WER: Claude improves on atis using a MiniMax-written memory, showing the memory is separate from its writer.The second frozen corrector reads a memory from a different corrector family.
  • Cross-corrector transfer: 3.94% vs. 6.08% WER: Claude’s self-formed memory on atis beats both the MiniMax memory and the n-best oracle.The self-formed memory reaches ρ=1.28 under the same validation-gated loop.
  • Transfer conditions: High-WER atis benefits from transfer, while memory is neutral to slightly harmful on clean low-WER domains with little recoverable headroom.The learned rules are described as general restraint rather than a domain-specific fix, supporting reuse across models.
  • Beyond ASR: 13 to 32% of oracle headroom: Voice Memory improves BLEU over the 1-best in every tested X→En speech-translation direction.The identical Voice Memory loop is applied across four translation directions.
  • Memory provenance: 7.17% weighted WER: Claude Sonnet 4.6’s memory beats auto-generated (7.52), expert hand-crafted (7.43), and expert-seeded (7.32) memories.Its 5.4 KB file is smaller than the 9.0 KB auto-generated and 7.6 KB expert-seeded variants.

6. Robust Voice Memory for Noisy ASR

Voice Memory extends its restraint policy to noisy ASR by storing noise-specific priors in text memories while keeping inference weights frozen. It matches RobustGER-style robustness on degraded corpora and improves difficult noise conditions while abstaining when little recoverable error exists.

  • Robust Voice Memory for Noisy ASR: Voice Memory optimizes one semantic-gated memory per noise family and reads it at inference without weight updates.The evaluation covers CHiME-4, VoiceBank-DEMAND, LibriSpeech+FreeSound, and NOIZEUS.
  • Robust Voice Memory for Noisy ASR: VoiceBank-DEMAND 1-best WERs match RobustGER within 0.1 point: party 22.59 vs. 22.6 and baby 7.98 vs. 8.0.This establishes comparability between the text-memory and fine-tuned RobustGER setups.
  • Robust Voice Memory for Noisy ASR: 7.75→7.31: Voice Memory lowers CHiME-4 dev-real WER, whereas static GER raises it from 7.75→8.13.The Voice Memory result has 𝜌=0.31.
  • Robust Voice Memory for Noisy ASR: 14.51→12.60: Voice Memory recovers WER on NOIZEUS at 5 dB, with 𝜌=0.36.The gain appears on genuinely degraded corpora where static GER over-corrects.
  • Robust Voice Memory for Noisy ASR: Voice Memory preserves near-null behavior on easy conditions, with party 𝜌=0.03, correctly abstaining when there is nothing to recover.The passage characterizes this as the same restraint policy holding under noise, without fine-tuning or a latent noise vector.

7. Analysis: Meaning versus Surface Error

The analysis finds that optimizing meaning rather than surface error improves correction modestly but consistently, because surface mismatches often preserve meaning. Voice Memory’s remaining errors separate into unrecoverable acoustic confusions and annotation artifacts where restraint prevents harm.

  • Semantic versus WER gating: The semantic gate wins on WER across both domains and seeds, with a modest ~0.2-point margin that is consistent in direction and more stable to optimization seeds.It rewards restraint by withholding credit for meaning-preserving edits that a WER gate would accept.
  • Surface error versus meaning: A decoupling slope of 0.38 to 0.60 means a unit of WER moves meaning by only ~half a unit.The slope is below 1 in every domain, indicating that surface error and semantic change are distinct quantities.
  • Surface error versus meaning: 53 to 68% of residual error mass preserves meaning, showing that most remaining surface error is semantically benign.This benign mass is measured using sentence-embedding semantic distance rather than asserted from text alone.
  • Residual error analysis: Voice Memory’s residual errors comprise unrecoverable acoustic confusions and annotation artifacts that should not be corrected.Acoustic confusions lack the needed token in the n-best list, while restraint specifically prevents damage from spelling and spacing variants.

8. Conclusion

Voice Memory improves a frozen corrector through a small, auditable text memory rather than weight changes. Its learned policy is restraint—deciding when to act or abstain—and it transfers across model families while targeting recoverable headroom.

  • Conclusion: Voice Memory improves a frozen corrector without changing any weights, storing everything learned in a small, auditable inference-time text memory.The file is the only difference between the worse and better corrector.
  • Conclusion: The learned policy is restraint: deciding per utterance which tokens to leave alone rather than always performing sequence-to-sequence correction.Voice Memory frames correction as a decision to act or abstain.
  • Conclusion: The gain scales with recoverable headroom, and a memory written by one model family can improve a reader from another because explicit rules transfer across weights.Voice Memory acquires trust from inference-time feedback rather than backpropagation.

Limitations

The main analyses rely on one open-source corrector, while key measures depend on the sentence encoder; supplementary results and released protocols partially reduce these dependencies.

  • Model and measurement dependence: The main analyses use MiniMax-M3, a 428B-parameter model with 23B activated parameters.Appendix B reproduces the core effect with an open Qwen3-30B-A3B corrector across ten datasets.
  • Model and measurement dependence: The ρ and benign-mass measures depend on the chosen sentence encoder.The study reduces this dependence using a small public on-device model with 610 ms average system latency and by releasing the protocol and audio-visual memory formation.

Ethics Statement · A. Reproducibility and Hyperparameters

Voice Memory’s human-readable memory supports auditing and bias review, while its reproducibility protocol specifies fixed optimization defaults, disjoint splits, single test reporting, and on-device semantic scoring.

  • Ethics Statement: Human-readable memories make auditing and bias review easier than inspecting weight updates.The artifact exposes learned behavior for inspection.
  • Ethics Statement: Domain memories may encode dataset-specific conventions, including spelling norms, that require review before transfer across dialects or populations.The passage identifies cross-population transfer as a potential concern.
  • Ethics Statement: Benign-mass analysis can surface encoded conventions before deployment.It is presented as one method for examining these conventions.
  • A. Reproducibility and Hyperparameters: 4 epochs and rollout batch 24 are the default optimization settings.These defaults are part of the reported reproducibility configuration.
  • A. Reproducibility and Hyperparameters: Each domain uses three disjoint splits, and test results are reported once.This defines the split and reporting protocol.
  • A. Reproducibility and Hyperparameters: All semantic scoring runs on-device.The passage specifies the execution setting for semantic scoring.

B. Broad Study with an Open Corrector … E. Qualitative Case Studies

Across broad ASR, corrector transfer, translation, and qualitative cases, Voice Memory improves over 1-best recognition by learning restrained, auditable interventions through text-only memory. Its gains are strongest when domain vocabulary or recoverable oracle headroom exists, while the method avoids regressions and can suppress harmful corrections.

  • B. Broad Study with an Open Corrector: The open-corrector evaluation reproduces the depth-study effect using Whisper-v2-Large as frozen decoder and Qwen3-30B-A3B as an inference-only corrector with per-domain memory.md.The corrector activates approximately 3B of its 30B parameters per forward pass, and no weights are updated.
  • B. Broad Study with an Open Corrector: 7.47% weighted WER is achieved by Voice Memory versus 8.36% for the raw 1-best across ten benchmarks, with no dataset below its baseline.ATIS improves from 8.40% to 3.00%, while CHiME-4 improves from 12.69% to 10.28%; CORAAL and SWBD change little.
  • B. Broad Study with an Open Corrector: 0.14 weighted-WER points come from negative rules alone, making restraint the load-bearing component by preventing rewrites of already-correct tokens.The ablation moves weighted WER from 7.66 to 7.52; few-shot examples add a further 0.05.
  • B. Broad Study with an Open Corrector: 42.9% of residual LS-Clean errors are forgivable as proper-noun or spelling-variant differences, yielding a practical hard floor of 0.749%.These error classes are structurally unreachable for a text-only corrector without the source document.
  • C. Corrector-Agnostic Memory: a Claude Case Study: A memory written by MiniMax transfers to an independent frozen corrector without fine-tuning, demonstrating that the portable asset is the text memory rather than model weights.The illustrative cross-corrector runs use 50 to 100 utterances and a sample-internal 𝜌 measure.
  • D. X→En Voice Memory: a Translation Mini-Study: The translation study applies the identical Voice Memory loop with a fixed language model serving as both corrector and BLEU-gated optimizer across four language directions.The task reconciles n-best English translations from non-English source utterances into a single translation scored by BLEU.
  • D. X→En Voice Memory: a Translation Mini-Study: Voice Memory improves BLEU over the 1-best in every tested X→En direction and recovers 13 to 32% of oracle headroom.The memory also beats memory-free reconciliation in three of four languages, while over-application appears on fr when reconciliation alone is already strong.
  • E. Qualitative Case Studies: Eight qualitative cases show Voice Memory fixing genuine errors, suppressing harmful static-GER edits, and generating transcripts absent from every n-best hypothesis.The cases span four ASR domains and include recovery of spelled-out forms such as s b / b n from sb / be.
Loading 2607.26410v1…