Source-linked AI summary
One Mean Log-Mel Vector per Phoneme: Simple and Fast Text Injection for Speech Recognition
Albert Zeyer, Davyd Naveriani, Dorian Koch, Ralf Schlüter, Hermann Ney
TL;DR
Transcribed audio is scarce compared with text, and text injection offers a way to train speech recognizers on unpaired text. This paper replaces frozen-TTS synthesis with phoneme-level lookup-table inputs derived from alignments. The approach trains 1.8 times faster than neural TTS while achieving comparable word error rates and a 12% relative WER reduction on LibriSpeech.
Problem
Text corpora are far larger than transcribed speech, but conventional use of text is limited to an external language model rather than direct end-to-end training.
Source: §1 · p. 1Method
Results
The lookup-table method trains 1.8 times faster than neural TTS while matching its WER, and yields a 12% relative WER reduction on LibriSpeech.
Source: §3.2 · p. 2, §4 · p. 4Takeaways & Limitations
Duration statistics are the most critical synthetic-input component, while the specific acoustic features matter less.
Source: §4 · p. 4The best results rely on a phoneme lexicon, and generating robust inputs directly from subword alignments requires further investigation.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Text injection trains speech recognition models on unpaired text alongside paired audio. We feed the speech encoder synthetic input derived from text so the whole model learns from it. Previously, we synthesized this input with a frozen text-to-speech (TTS) model. Here, a simple lookup table replaces the TTS model: a mean log-mel vector (or one-hot encoding) per phoneme, repeated for a random duration drawn from alignment statistics, with silence between words. Training is 1.8 times faster than with the neural TTS, achieving comparable word error rates. With an external language model, our system reaches 1.36 / 2.90 / 1.52 / 3.24 % WER on LibriSpeech {dev,test}-{clean,other}, to our knowledge state-of-the-art using only LibriSpeech paired data, text, and lexicon. We analyze components of synthetic input (durations, acoustic features), compare different HMM aligners, evaluate scaling on the 25 000 h Loquacious dataset, and show that full-encoder injection yields strong relative improvements compared to partial-encoder and decoder methods. The code of the whole pipeline behind every number in this paper is public: github.com/rwth-i6/returnn-experiments/tree/master/2026-text-injection
1. INTRODUCTION
Because transcribed audio is scarce relative to text, text injection trains the end-to-end recognizer on unpaired text by feeding synthetic inputs to the speech encoder. This work replaces the prior frozen-TTS synthesis with a per-phoneme lookup table, reporting faster training and strong results under constrained data conditions.
1. INTRODUCTION
- Text injection trains the end-to-end model directly on unpaired text, with synthetic text-derived input entering at the speech-encoder input.This differs from the traditional use of text for an external language model in search; the model, losses, and search remain unchanged.Source: §1 · p. 1
- Neural TTS synthesis had outperformed prior implicit or explicit matching methods, but its computational expense motivates the simpler lookup-table alternative.The lookup-table input follows the audio path with SpecAugment and needs no text encoder, masking, or extra losses.Source: §1 · p. 1
- Earlier textogram methods feed symbol sequences to the encoder, while this approach uses acoustic mean log-mel vectors per phoneme with alignment-derived durations.The paper states that alignment-derived durations perform best, while fixed-duration character textograms retain 60% of its gain.Source: §1 · p. 1
- The lookup-table approach replaces neural TTS and accelerates training by 1.8× while achieving a 12% relative WER reduction on LibriSpeech.The paper also reports that full-encoder injection yields highly competitive relative improvements over partial-encoder or decoder methods.Source: §1 · pp. 1–2
2. METHOD
The method converts lexicon-mapped text into synthetic acoustic input using phoneme-level mean log-mel vectors and sampled durations. It inserts inter-word silence and feeds the result through the audio front end alongside paired audio during training.
2. METHOD
- The method derives a mean log-mel vector and median duration for each phoneme from alignments of paired audio, then maps text sentences to phonemes through a lexicon.The duration is sampled as round(m · s · exp(σε)), with scale s and jitter σ.Source: §2 · p. 2
- It inserts silence between words, repeats each phoneme vector for its sampled duration, and linearly interpolates at boundaries before applying the audio front end and SpecAugment.Source: §2 · p. 2
- Audio and text sentences are randomly mixed in batches and trained with the same losses.Source: §2 · p. 2
3. EXPERIMENTS
The experiments compare lookup-table text injection with audio-only and TTS baselines, then test its aligners, duration and feature choices, data requirements, and scaling. Text injection improves LibriSpeech WER, while experiments on Loquacious show that its relative gain depends on the text-to-audio ratio and paired-data scale.
3.1. Setup
- LibriSpeech experiments use 960 h of paired audio and 800M-word injection text; Loquacious experiments use transcripts from its 25,000 h set.The Loquacious subsets span 250 h to 25,000 h of paired audio.Source: §3.1 · p. 2
3.2. Results
- 12% relative WER reduction on LibriSpeech test-other (4.16 to 3.66); with an external LM, test-other WER falls from 3.38 to 2.93.The lookup table matches GlowTTS WER while training 1.8 times faster (130 h vs. 235 h on 4 GPUs).Source: §3.2 · p. 2, Table 1 · p. 2
- 1.8 times faster training (130 h vs. 235 h) accompanies a self-contained single-Gaussian HMM that retains most of the gain (3.46 vs. 3.40 WER).The HMM uses three states per phoneme and one diagonal Gaussian per state; its state means form the log-mel table.Source: §3.2 · p. 2, §3.1 · p. 2, Table 2 · p. 2
- Scaling durations to 0.7× (5.9 frames per phoneme versus the aligned 8.4) performs best at 3.66 test-other WER; fixed six-frame durations score 3.73.The log-mel table is robust to temporal compression; randomly scaling GlowTTS durations by 0.2–0.5 degraded test-other by roughly 0.5 percentage points, though the comparison is not exactly matched.Source: §3.2 · p. 2, Table 3 · p. 3
- Trainable embeddings (3.47) and one-hot channels (3.37) perform similarly to mean log-mel features (3.40); character units score 3.58 versus the 3.85 baseline.Uniform durations cost the log-mel table 0.13 WER, while removing interpolation or inter-word silence has negligible impact.Source: §3.2 · p. 3, Table 4 · p. 3
- A 1:1 sampled text-to-audio word ratio is optimal, but halving paired audio severely degrades WER; random batch order beats fixed interleaving by 0.08.For TTS injection, mixing modalities in one batch outperforms separate batches by 1.4 percentage points absolute WER.Source: §3.2 · p. 3, Table 5 · p. 3, Table 6 · p. 3
- With fixed 25,000 h Loquacious text, relative gains over audio-only shrink from 24% at 250 h paired audio (100:1 ratio) to 3% at 25,000 h (1:1).At matched compute, text injection matches or exceeds extended audio training.Source: §3.2 · pp. 3–4, Table 8 · p. 4
- On Loquacious, mean log-mel source matters little, uniform sampling helps YODAS, and more text improves test performance.With the Loquacious LM in search, CTC+AED+LM WER is 6.17 vs. 5.83 on dev and 6.97 vs. 6.73 on test.Source: §3.2 · p. 4, Table 10 · p. 4, Table 9 · p. 4
4. CONCLUSIONS
A phoneme-level lookup table with alignment-derived durations replaces neural TTS for text injection, accelerating training while improving LibriSpeech WER. The conclusion highlights duration statistics, lexicon dependence, scaling behavior, and full-encoder injection as important findings.
4. CONCLUSIONS
- A 12 % relative WER reduction over the baseline reaches 1.36 / 2.90 / 1.52 / 3.24 % with a large model on LibriSpeech.The conclusion reports these as the method's LibriSpeech results.Source: §4 · p. 4
- Replacing neural TTS with a lookup table requires no synthesizer and makes training 1.8 times faster.The table uses durations from a phoneme alignment.Source: §4 · p. 4
- Duration statistics are the most critical synthetic-input component, while the specific acoustic features matter less.Source: §4 · p. 4
- The best results rely on a phoneme lexicon; characters retain 60 % of the gain without one, while robust inputs from subword alignments need further investigation.Source: §4 · p. 4
- On Loquacious, relative gains scale predictably with text-to-audio ratio and paired-data amount, and acoustic features generalize across datasets without retraining.Source: §4 · p. 4
- At equal scale, full-encoder text injection generally outperforms injection into a shared upper encoder or the decoder alone.This comparison is based on relative gains across the literature.Source: §4 · p. 4