WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Small Pretrained Speech Recognition Transformers

• Target-domain parlance can include unseen words, while costly or infeasible speech collection and small-model deployment constrain adaptation options.Source: §1 · p. 1
• Language-model fusion and internal language-model estimation adjust decoding probabilities, but may fail on unseen words because end-to-end ASR poorly generalizes to novel grapheme sequences.Source: §1 · p. 2
• TTS-based fine-tuning provides input–output supervision but does not explicitly guide how an end-to-end model’s internal state adapts to a target domain.Source: §1 · p. 2
• WhisTLE trains a VAE to model ASR encoder outputs from text, then fine-tunes the decoder with a text-only encoder, optionally alongside TTS adaptation.Source: Figure 1 · p. 1, §1 · p. 2
• WhisTLE is presented as the first deeply supervised, text-only adaptation method for pretrained encoder–decoder ASR transformers, including non-LLM models.Source: §1 · p. 2
• With TTS, WhisTLE achieved a 49.0% average relative WER reduction over no adaptation and outperformed non-WhisTLE baselines in 100 of 112 scenarios.Source: §1 · p. 2
• The authors recommend including WhisTLE in standard practices for adapting ASR models to new domains.Source: §1 · p. 2

2.1 Our WhisTLE Approach

• WhisTLE models transformer encoder representations from text rather than audio or Mel spectrograms, targeting existing pretrained ASR models and avoiding extra inference resources.The approach rests on hypotheses that latent-state supervision helps domain adaptation and that modeling encoder outputs is more training-efficient than modeling speech.Source: §2 · p. 2, §2.1 · p. 3
• During text-only adaptation, the frozen TLE approximates Whisper encoder outputs from transcript tokens and supplies them to the decoder for log-likelihood fine-tuning; the TLE is discarded for standard inference.This preserves the pretrained encoder–decoder model at inference without additional run-time or memory requirements.Source: §2.1 · p. 3
• The TLE is a convolutional VAE with three encoder layers, four decoder layers, residual connections, and a transposed convolution that upsamples text embeddings.Its VAE is trained on source-domain speech–text pairs, using Whisper encoder outputs as targets.Source: Figure 2 · p. 3, §2.1 · p. 4
• The VAE objective combines masked and unmasked mean-squared reconstruction losses with a KL term; separate reconstruction terms address padding and speech portions of Whisper outputs.The unmasked reconstruction loss is scaled by λ, and the KL term by β.Source: §2.1 · p. 4

3 Experimental Setup

• The six datasets comprise CommonVoice and LibriSpeech in-domain, plus EMNS, EmoV-DB, ST-AEDS, and EABI out-of-domain.The out-of-domain sets vary in emotional intensity, accent, and recording environment.Source: §3 · p. 4
• WhisTLE is evaluated by fine-tuning Whisper in-domain, training the VAE on the same data, adapting text-only on out-of-domain data, and testing on its audio and text pairs.Baselines include TTS-generated data, shallow fusion, deep fusion, and internal language model estimation; the methods are also combined.Source: §3 · p. 4
• Text-only training runs for 50K steps at batch size 8, with two in-domain audio-text training steps per step to prevent catastrophic forgetting.In-domain fine-tuning runs for 100K steps at batch size 8.Source: §3 · p. 4
• The language-model baselines use trigram LMs: shallow fusion combines log probabilities, deep fusion uses a bottleneck network, and ILME subtracts the ASR model’s internal-LM output.Fusion weights are selected from tested values; ILME estimates internal-LM probabilities by decoding silence.Source: §3 · p. 5
• TTS adaptation synthesizes out-of-domain audio using randomly selected in-domain speakers, then fine-tunes the base model on that audio; combined TLE uses the same training data.FastSpeech2 and SpeechT5 are used for CommonVoice and LibriSpeech, respectively.Source: §3 · p. 5
• Figure 4 summarizes average WER changes across out-of-domain datasets when TLE is added to treatment combinations; Table 1 gives examples comparing TLE, TTS, and their combination.The figure reports paired-bootstrap significance at the 95% level.Source: Table 1 · p. 4, Figure 4 · p. 6

4.1 Main Whisper Results

• 5.6 average WER for TLE+TTS was best across treatments, versus 7.2 for TTS and 11.9 without adaptation; adding TLE helped 69 of 80 scenarios, with a 22% average relative WER drop.The reported differences were statistically significant at the 95% level according to paired bootstrap testing.Source: §4.1 · p. 6
• TLE+TTS gains averaged 3–4 WER on ST-AEDS and EMNS versus either method alone, 2 on EABI, and 1.5 on EmoV-DB; improvement direction was robust across Whisper sizes.Whisper-medium had average WER below 6.0 on all out-of-domain sets; Whisper-large’s LibriSpeech results were noisier.Source: §4.1 · p. 6
• Across treatments, TLE reduced WER in 116 of 144 (81%) cases; adding it gave relative improvements of 23% with no adaptation, 15% with TTS, and 14% with ILME.Of 28 cases without improvement, 15 were already worse than baseline before TLE was added, and all 15 involved external language modeling.Source: §4.1 · pp. 6–7

4.2 External Language Model-Based Treatments

• ILME lowered average out-of-domain WER by 4.3 with CommonVoice-trained models but raised it by 8.0 points with LibriSpeech; shallow and deep fusion increased WER by 13.2 and 17.4, respectively.The authors attribute shallow-fusion failures mainly to increased hallucinations and deep-fusion failures largely to a fusion network trained on in-domain audio that was insufficient for out-of-domain audio.Source: §4.2 · p. 7

4.3 Linear Mixed-Effects Analysis

• The mixed-effects model found significant TLE WER reductions for Whisper-large (-5.7 points) and Whisper-medium (-3.4 points), with no statistically significant interaction terms.The model used 500 speech–text examples per out-of-domain dataset; shallow and deep fusion were omitted because of frequent hallucination.Source: Table 4 · p. 7, §4.3 · pp. 7–8

4.4 Auxiliary Canary Experiments

• With LibriSpeech as the in-domain corpus, Canary-1B reduced relative WER by 16.3% and Canary-180M-flash by 71%, improving on all four out-of-domain datasets.With CommonVoice, gains were smaller: 4.4% for Canary-1B and 6.0% for Canary-180M-flash.Source: Table 5 · p. 8, Table 6 · p. 8, §4.4 · p. 8

5 Related and Future Work

• Earlier generative approaches synthesized out-of-domain speech or Mel spectrograms; the paper’s TLE models ASR encoder representations from text and is described as faster to train than the compared TTS systems.The reported TLE sizes were 91M parameters for Whisper-medium and 104M for Whisper-large, with 100K training steps and batches of four sentences.Source: §5 · p. 8
• Future studies could characterize efficiency–effectiveness trade-offs, predict which adaptation combinations work best, and examine how audio-derived and text-predicted latent encodings differ.The observed improvement in homonyms and named entities from combining TLE and TTS is described as anecdotal and needing further study.Source: §5 · p. 8
• External language-model approaches include shallow fusion, internal-LM estimation, reranking, and deep fusion; speech LLM scale and inference overhead motivate alternatives for deployment.TLE and TTS add no inference-time overhead, while n-gram models add minimal overhead relative to loading an LLM.Source: §5 · pp. 8–9
• Other architecture-based methods enable text-only training through modified ASR models, modular internal LMs, or adaptation branches [Deng and Woodland, 2023a] [Meng et al., 2023].The authors propose future generalization beyond speech recognition to other encoder–decoder tasks.Source: §5 · p. 9

6 Conclusions

• WhisTLE models encoder outputs from text and fine-tunes the decoder, complementing TTS input–output supervision with latent supervision.The method is presented as text-only adaptation for pretrained encoder–decoder ASR.Source: §6 · p. 9
• With TTS, WhisTLE reduced WER by a relative 49.0% on average and outperformed all non-WhisTLE baselines in most scenarios.The authors report WER improvements both when WhisTLE is used alone and when combined with other adaptation approaches.Source: §6 · p. 9

Limitations

• WhisTLE requires a sufficiently large decoder; models without a traditional decoder, such as CTC systems, may not be good candidates.The authors note that wav2vec 2.0 is an example of a CTC system.Source: Limitations · p. 9
• Evaluation datasets cover English speakers in the US and UK, so effectiveness on accents diverging further from those varieties remains unestablished.The authors state that further testing would be required.Source: Limitations · p. 9
• TLE training requires access to some in-domain audio-text data, despite the method's text-only adaptation setting.The method is trained jointly with in-domain audio-text pairs and out-of-domain text data.Source: Limitations · p. 9
• The external language-model comparison is limited to an n-gram model; billion-scale models might bolster the tested fusion and ILME approaches.This limit reflects the authors' constraint on inference speed and cost.Source: Limitations · p. 9

A Training Details

• Whisper-large and Whisper-medium TLE adapters used batch size 4 for 100k steps, with learning rates of 1.25e-5 and 2.5e-5, respectively.Training took approximately 12 hours for Whisper-large and 7 hours for Whisper-medium on an Nvidia L40S GPU.Source: A Training Details · p. 11
• Whisper-large and Whisper-medium ASR experiments used equivalent batch size 8 for 100k steps, with learning rates of 6.25e-8 and 1.25e-7, respectively.The equivalent batch sizes came from batch sizes of 2 with 4 gradient accumulation steps for Whisper-large and 4 with 2 steps for Whisper-medium.Source: A Training Details · p. 11
• Canary results were included to demonstrate TLE's applicability beyond Whisper, not to provide an in-depth adaptation comparison.The authors did not run the full range of Canary experiments because of computational requirements.Source: A Training Details · p. 11
• Each result in the main results tables is from a single run with seed 33, because computational requirements prevented multiple trials for all experiments.The statement covers Tables 2, 3, 5, and 6.Source: A Training Details · p. 11
• Whisper analysis required 48 training runs, excluding deep-fusion training, TTS audio generation, and TLE module training; OOD audio generation took approximately 6 hours with FastSpeech2 and 5.5 hours with SpeechT5.The generation times refer to audio for the out-of-domain data.Source: A Training Details · p. 11