PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowEdit: Associative Memory for Lifelong Pronunciation Adaptation in Flow-Matching TTS

Harshit Singh, Ayush Pratap Singh, Nityanand Mathur

arXiv:2606.20518v1cs.AI

TL;DR

Deployed flow-matching TTS systems can retain proper-noun pronunciation errors unless retrained. FlowEdit optimizes latent text-conditioning edits and stores them in a gated Hopfield memory; it reports a 92.7% relative reduction in target-word PER while preserving general-speech quality.

  • Problem

    Frozen flow-matching TTS systems lack a post-deployment way to correct persistent proper-noun pronunciation errors without retraining.

    Source: §1 · p. 1

  • Method

    FlowEdit optimizes token-level perturbations in text embeddings from corrective audio feedback, then stores them in a Hopfield memory for gated retrieval.

    Source: §1 · p. 1, §3.2 · p. 2

  • Results

    92.7% relative reduction in target-word PER versus the zero-shot baseline, with general PER on LibriTTS-R remaining 4.1%, statically tied to baseline.

    Source: §4.2 · p. 3

  • Takeaways & Limitations

    FlowEdit provides lifelong pronunciation adaptation without weight updates, with corrections reported to take about 15 seconds each.

    Source: §5 · p. 4

    Residual PER is higher for tonal languages because the mel reconstruction loss underweights F0 relative to the spectral envelope.

    Source: §4.5 · p. 4

Abstract

from arXiv · show

Flow-matching text-to-speech systems achieve remarkable zero-shot quality but remain static after deployment: pronunciation errors on out-of-vocabulary proper nouns persist unless the model is retrained. We introduce FlowEdit, a life-long adaptation framework for frozen flow-matching TTS that learns pronunciation corrections as latent conditioning edits rather than weight updates. When corrective feedback is provided, FlowEdit optimizes a token-level perturbation in the text embedding space, then stores the correction in a Modern Hopfield Network serving as content-addressable episodic memory. At inference, corrections are retrieved via soft attention with a similarity gate, enabling fuzzy morphological matching. On our curated benchmark of 312 multilingual proper nouns across 18 language families, FlowEdit reduces target-word Phoneme Error Rate by 92.7% relative to the zero-shot baseline while maintaining identical general-speech quality. Corrections complete in approximately 15 seconds on a single GPU.

1. Introduction

FlowEdit addresses persistent proper-noun and loan-word mispronunciations in deployed, frozen TTS systems without retraining. It optimizes edits to text-conditioning embeddings and stores them in a Hopfield Network for retrieval during inference.

1. Introduction

  • FlowEdit corrects pronunciation by optimizing token-level perturbations to text conditioning instead of updating the frozen DiT’s weights.Optimized edits are stored in a Modern Hopfield Network and retrieved via soft attention with a similarity gate, supporting fuzzy morphological matching.Source: §1 · p. 1
  • The framework targets persistent proper-noun and foreign loan-word errors that remain in deployed TTS systems without costly retraining.The introduction contrasts FlowEdit with G2P dictionaries that fail on some polyglot names, fine-tuning that risks forgetting and voice drift, and weight editing that risks cumulative interference.Source: §1 · p. 1
  • 92.7% relative target-word PER reduction is reported alongside mathematically guaranteed zero forgetting of general speech.The paper also introduces POLYGLOT-NOUNS, a benchmark for personalized pronunciation adaptation.Source: §1 · p. 1

2. Related Work

Prior work offers high-quality zero-shot TTS and several adaptation approaches, but the cited systems lack post-deployment pronunciation correction suited to frozen TTS. FlowEdit instead uses audio feedback and external memory to adapt without editing model weights.

2. Related Work

  • Recent TTS systems deliver zero-shot quality, but the cited systems offer no mechanism for post-deployment pronunciation correction.G2P dictionaries and lexicons are difficult to apply when end-to-end models map raw text or BPE directly to spectrograms; phoneme injection can require users to provide IPA.Source: §2 · p. 1
  • NLP editing and continual-learning methods can require prior training data or architectural expansion, unavailable for frozen deployed TTS systems.The section discusses weight-editing methods, regularization, lateral connections, pruning, and replay buffers; it notes that weight edits risk collateral damage as edits accumulate.Source: §2 · p. 1
  • FlowEdit addresses the gap with audio-feedback pronunciation correction and external Hopfield memory rather than weight editing.The paper says this avoids catastrophic forgetting and parameter drift.Source: §2 · pp. 1–2

3. Methodology

FlowEdit uses the differentiable inference path of a frozen flow-matching TTS model to optimize text-conditioning embeddings, then stores corrections in a Hopfield Refiner. Its gated retrieval supports fuzzy matching while the TTS backbone remains frozen.

3.1. Preliminaries: Conditional Flow Matching

  • The conditional flow-matching model synthesizes speech by integrating a learned ODE from t=0 to t=1, and its differentiable conditioning path permits embedding optimization without changing model parameters.The model uses a fixed-step Euler solver with N=32 steps in the reported setup.Source: §3.1 · p. 2

3.2. The FlowEdit Framework

  • Correction begins with reference audio and target text; forced alignment identifies target token indices, expanded by one token on each side.The DiT parameters remain frozen while a zero-initialized perturbation is optimized for the text embeddings.Source: §3.2 · p. 2
  • The latent optimization masks non-target positions and uses an adjoint sensitivity method to recover conditioning gradients with a reverse-time ODE.The objective regularizes the perturbation, with λ=0.001; the adjoint state is defined as a(t) = ∂L/∂xt.Source: §3.2 · p. 2
  • The correction loop stores each finalized correction in a Hopfield Refiner after optimizing the latent edit.Figure 1 depicts reference audio triggering forced alignment, perturbation optimization, and a memory write; the inference path refines encoded text before decoding with the frozen DiT.Source: Figure 1 · p. 2, §3.2 · p. 2
  • Hopfield retrieval uses soft attention and a similarity gate to suppress irrelevant matches and support fuzzy morphological matching.The paper gives “Linux” and “Linux’s” as an example; duplicate corrections are consolidated, and memory is bounded by LRU pruning at a user-defined budget.Source: §3.2 · p. 2
  • Context-conditioned keys use a Gaussian-weighted average of surrounding text embeddings within a ±3-token window for homograph disambiguation.The example contrasts “bass” as a fish with “bass” as a guitar.Source: §3.2 · p. 3

4. Experiments

Across POLYGLOT-NOUNS, FlowEdit improves target-word pronunciation while preserving general-speech quality, and the experiments examine component ablations, continual editing, transfer, scaling, and robustness.

4.1. Experimental Setup

  • POLYGLOT-NOUNS covers 312 proper nouns across 18 language families, paired with five carrier sentences each; evaluation also measures forgetting on 500 LibriTTS-R utterances.The benchmark includes 1,560 clips, with target-word PER reported using wav2vec 2.0 and human transcription as a check.Source: §4.1 · p. 3

4.2. Main Results

  • 92.7% relative reduction in target-word PER from the zero-shot baseline; FlowEdit reaches 3.1% PERtarget while general PER remains 4.1%.Human transcription on the 60-word subset gives 3.4% PER, and FlowEdit outperforms full fine-tuning at 8.2% PERtarget.Source: Table 1 · p. 3, §4.2 · p. 3

4.3. Ablation Studies and Convergence

  • Removing Hopfield write-back degrades correction longevity to 6.9% PER, while removing gating raises general PER to 5.8%.Strict 1-nearest-neighbor indexing gives 3.8% PER; optimization descends rapidly before refining through step 50.Source: §4.3 · p. 3, Table 2 · p. 3, Figure 2 · p. 4

4.4. Continual Editing and Morphological Transfer

  • Across 200 edits, FlowEdit has 0.1 drift versus 6.8 for fine-tuning, and soft attention yields 8.4% PER on morphological variants versus 36.4% for rigid lexicons.The table covers long-horizon stability and transfer to unseen inflected variants.Source: §4.4 · p. 3, Table 4 · p. 4

4.5. Compute Scaling and Speaker Transfer

  • Correction runs take 42 seconds on an NVIDIA L4 and 38 seconds on an RTX 3090; retrieval overhead stays below 35 ms for at most 500 corrections.At 10,000 synthetic memory entries, PERtarget is 6.8% and retrieval latency is 112 ms per utterance.Source: §4.5 · pp. 3–4
  • A correction learned from one speaker transfers across 11 others: mean cross-speaker PER is 3.6% versus 3.1% for the source speaker, with no accent group above 4.2%.The panel has 12 speakers across four accent groups; performance degrades as synthetic memory size grows from 500 to 10,000 entries.Source: §4.5 · p. 4
  • Residual errors concentrate in single-phoneme monosyllables and tonal languages, where F0 RMSE is 18.3 Hz versus 4.7 Hz for non-tonal words.Adding an F0 loss reduces Mandarin PER from 9.1% to 6.4% and Vietnamese PER from 5.3% to 4.1%, at a cost of three optimization seconds.Source: §4.5 · p. 4

4.6. Latent Edit Geometry and Robustness

  • Edit vectors form vowel-remapping, consonant-change, and stress-shift clusters, with mean pairwise cosine similarity of 0.07; PER stays within 0.4% of baseline at SNR ≥15 dB.At 32 kbps Opus compression, PER is 3.2%.Source: §4.6 · p. 4

5. Conclusion

FlowEdit uses gradient-based latent optimization and Hopfield memory for lifelong pronunciation adaptation without changing model weights. It reports strong pronunciation correction, zero forgetting, and speaker-agnostic transfer, while identifying future extensions.

5. Conclusion

  • 92.7% relative PER reduction, with mathematically guaranteed zero forgetting of general speech.The conclusion also reports stability across 200 sequential edits.Source: §5 · p. 4
  • FlowEdit provides speaker-agnostic transfer and completes each correction in 15 seconds.The approach combines gradient-based latent optimization with Hopfield memory.Source: §5 · p. 4
  • Future work will integrate phoneme-conditioned generation and expand memory to capture domain-specific prosody.Source: §5 · p. 4

6. Use of Generative AI Disclosure

The authors used generative AI for language refinement and coding assistance. They state that the research contributions are their own and that they reviewed the assisted text and code and take responsibility for the paper.

6. Use of Generative AI Disclosure

  • Generative AI tools assisted with language refinement and writing and debugging experiment and analysis software.Source: §6 · p. 5
  • The authors state that the methodology, experimental design, results, and scientific claims are their own contributions.Source: §6 · p. 5
  • The authors reviewed and verified AI-assisted text and code and take full responsibility for the paper’s content.Source: §6 · p. 5
Loading 2606.20518v1…