PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Expert Routing for Text-Only Domain Adaptation of Factorized Transducer ASR

Shashi Kumar, Esaú Villatoro-Tello, Thibault Bañeras-Roux, Sergio Burdisso, Petr Motlicek, Kadri Hacioglu, Andreas Stolcke

arXiv:2610.32637v1cs.CL

TL;DR

Text-only adaptation of a factorized transducer can improve target-domain recognition while degrading source-domain performance, and one interpolation weight cannot suit every prefix. The paper proposes conservative residual routing, using a per-prefix gate to admit corrections from the adapted predictor while retaining the base as the output. On two held-out domains it retains essentially all text-only gains while leaving source-domain WER unchanged.

  • Problem

    Text-only adaptation can lower WER on target domains but raise it on the source domain, while one interpolation weight cannot accommodate prefixes where the adapted predictor helps and hurts.

    Source: §1 · p. 1

  • Method

    Conservative residual routing keeps the base non-blank predictor as the output and uses a learned per-prefix gate to admit the text-adapted predictor’s residual correction.

    Source: §1 · p. 1

  • Results

    Routing retains essentially all text-only gains on two held-out domains while leaving source-domain WER unchanged, a result no static interpolation weight matches across all three sets.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The gate does not act as a simple domain-level switch: as predictor disagreement grows, it responds differently on target-domain and source-like data.

    Source: §1 · p. 2, §6 · p. 4

    The gate has no access to acoustics, test-set identity, or either predictor’s absolute confidence; when predictors agree, its output is the same constant across domains.

    Source: §3.1 · p. 2

Abstract

from arXiv · show

Fine-tuning the non-blank predictor of a factorized transducer on target-domain text improves recognition on that domain but degrades performance on the source domain. A single interpolation weight between base and adapted predictors cannot resolve this: the adapted predictor helps at some prefixes and hurts at others, often within one utterance. We propose conservative residual routing: the adapted predictor becomes a residual correction to the base, and a 16k-parameter gate decides at each prefix how much of it to admit. Closing the gate reproduces the base exactly. On a 40-hour conversational corpus, routing retains essentially all of the text-only gain on two held-out domains (19.89 to 16.74, 20.91 to 16.21 WER) while leaving source-domain WER unchanged (12.90 to 12.93); the adapted predictor alone costs 1.08 WER. Routing needs no per-domain tuning on target-domain audio. Analysis reveals the gate does not behave as a simple domain-level switch: it modulates intervention through the disagreement between the two predictors, suppressing it in-domain and amplifying it out-of-domain.

1. INTRODUCTION

Text-only adaptation of a factorized transducer improves unseen target domains but can harm the source domain, and a single interpolation weight cannot adapt to prefixes where the expert helps or hurts. Conservative residual routing keeps the base predictor as the output and learns how much of the adapted predictor’s correction to admit.

1. INTRODUCTION

  • Factorized transducers provide a standalone non-blank predictor that can be fine-tuned on target text and then used in place of the original during decoding.Earlier approaches include internal-LM adaptation and joint training with unpaired text.Source: §1 · p. 1
  • Text-only adaptation lowers WER on two unseen domains but raises source-domain WER, while helpful and harmful prefixes can occur within one utterance.A single scalar interpolation weight must serve every prefix.Source: §1 · p. 1
  • The proposed policy keeps the base predictor as the output and admits a learned amount of the text-adapted predictor’s residual correction at each prefix.The gate mixes predictors in probability space and can be clamped shut to recover the base exactly.Source: §1 · p. 1
  • The gate is trained through the transducer objective on paired source data and supervised by the expert’s next-token margin advantage on target text, without target-domain audio.Source: §1 · p. 1
  • Routing retains essentially all text-only gains on two held-out domains while leaving source-domain WER unchanged, and no static interpolation weight matches it across all three sets.The gate’s behavior also varies with predictor disagreement rather than acting as a simple domain-level switch.Source: §1 · pp. 1–2
  • Unlike a simple domain-level switch, the gate responds differently to growing predictor disagreement across domains.Related approaches have different constraints: MoE routing lacks an exact fallback to the deployed model, while adaptive fusion uses a fixed rule over LM scores.Source: §1 · pp. 1–2

2. FACTORIZED TRANSDUCER

A factorized transducer separates blank and vocabulary prediction: the blank branch uses a joiner, while the non-blank distribution combines acoustic and internal-LM scores. Its non-blank predictor is a standalone language model that can be trained on text alone.

2. FACTORIZED TRANSDUCER

  • The transducer defines a distribution over alignments from blank and label emissions, with separate blank and non-blank predictors consuming label history.Source: §2 · p. 2
  • The blank probability comes from a joiner over encoder and blank-predictor states.Source: §2 · p. 2
  • The non-blank distribution is the softmax of summed acoustic and internal-LM scores.Source: §2 · p. 2
  • The non-blank predictor is a standalone language model that never sees the encoder and can be trained with cross-entropy on text alone.Source: §2 · p. 2
  • At decoding, internal-LM fusion weights α and β combine scores in the transducer decoding rule.Source: §2 · p. 2

3. CONSERVATIVE RESIDUAL ROUTING

Conservative residual routing keeps the base predictor as the output and uses a per-prefix gate to admit a text-adapted predictor’s residual correction. The gate is trained on paired source data and target text, with losses that encourage recognition while limiting unnecessary intervention.

3.1. Residual trust gate

  • A per-prefix gate scales the expert’s residual correction to the base distribution, producing a valid mixed distribution without renormalization.The gate reads the difference between base and expert distributions, and the realized intervention depends on both gate trust and predictor disagreement.Source: §3.1 · p. 2

3.2. Training on paired source audio-text

  • On paired source data, the transducer and internal-LM losses train the gate for recognition, while KL and sparsity regularizers discourage deviation from the base.Only the gate is updated; the recognition loss provides a reason to intervene when the expert’s correction helps on source data.Source: §3.2 · p. 2, §3.3 · p. 3

3.3. Linguistic advantage from target text

  • On target text without audio, the gate is supervised by the expert’s next-token margin advantage, a proxy for whether its benefit exceeds the cost of departing from the base.The margin measures whether the expert improves the reference token’s margin over its strongest competitor relative to the base.Source: §3.3 · pp. 2–3

4. EXPERIMENTAL SETUP

The experiments use a factorized transducer fine-tuned on paired call-center audio and text, then adapt its non-blank predictor on text from two unseen domains. Evaluation compares routed systems with static interpolation on two out-of-domain sets and one in-domain set.

Base model

  • The base model is a 69.8 M-parameter Zipformer factorized transducer, fine-tuned on 40 hours of call-center conversations after LibriSpeech training.Its non-blank predictor has 4.7 M parameters and a 500-unit BPE vocabulary.Source: §4 · p. 3

Target text and expert

  • Target text comprises 10,981 retail and 18,889 Telco utterances, with no target-domain audio used to train or select routed systems.The expert is a copy of the non-blank predictor fine-tuned on this text for five epochs.Source: §4 · p. 3

Gate

  • The only trainable module is a 16,033-parameter gate, selected by target-text development loss; static interpolation instead uses target-audio development WER.The gate has 0.02% of the full model’s parameters and 0.3% of the predictor’s.Source: §4 · p. 3

Evaluation

  • Evaluation reports WER on Retail and Telco out-of-domain sets and an in-domain set, using modified beam search with beam size 4.Table 1 covers WER (%) on the two out-of-domain test sets and the in-domain test set.Source: §4 · p. 3, Table 1 · p. 3

5. RESULTS

Text-only adaptation improves held-out-domain recognition but regresses on the source domain; residual routing retains most target gains while preserving source-domain performance. The gate responds to predictor disagreement differently in-domain and out-of-domain, rather than acting as a simple domain switch.

5.1. Baseline parity

  • 3.05/7.40 WER for the factorized transducer versus 3.08/7.51 for a matched standard transducer on LibriSpeech test-clean/test-other, with internal-LM fusion recovering the small factorization loss.With α=1, β=0, the factorized transducer reaches 3.12/7.67 WER.Source: §5.1 · p. 3

5.2. Text-only adaptation and routing

  • 15.1% and 23.0% relative held-out-domain WER improvements from text adaptation are largely retained by routing, which performs the same as the base in-domain while recovering most out-of-domain gains.Text-only adaptation alone costs 1.08 WER in-domain, an 8.4% relative regression.Source: §5.2 · p. 3
  • Without the KL anchor, Retail/Telco WER improves from 16.86/16.57 to 16.74/16.21 while source-domain WER changes only from 12.91 to 12.93.Source supervision and sparsity remain active in this ablation.Source: §5.2 · p. 3
  • Increasing λsp to 1.0 reaches 12.80 source-domain WER while reducing target gains; removing sparsity changes WER only slightly relative to default.The ablations do not isolate source supervision's contribution.Source: §5.2 · p. 4
  • A constant γ=0.4 preserves source-domain performance but is 2.4% and 5.0% relatively worse on Retail and Telco than routing, and requires target-domain audio for selection.No swept constant is better than routing on all three sets at once.Source: §5.2 · p. 4

5.3. What the gate learns

  • Median trust is nearly identical across test sets—0.466 on both out-of-domain sets and 0.465 in-domain—so trust distributions alone do not indicate a domain-level switch.Mean trust is 0.49, 0.50, and 0.44, respectively.Source: §5.3 · p. 4
  • At matched disagreement, trust rises on out-of-domain sets but falls in-domain; the gate responds to the direction of disagreement without a domain label.Trust starts at 0.466 where predictors agree, then separates as disagreement grows.Source: §5.3 · p. 4, Figure 2 · p. 4
  • For states with D ≥0.7, mean trust, intervention, and agreement with the expert's top choice are each roughly twice as high out-of-domain as in-domain.Most utterances in every set lie between the two trust extremes.Source: §5.3 · p. 4

6. CONCLUSION

Conservative residual routing uses a per-prefix gate to admit a text-adapted predictor's correction to the base. It retains nearly all text-only gains on two held-out domains while leaving source-domain WER unchanged, without target-domain audio tuning.

6. CONCLUSION

  • A 16k-parameter per-prefix gate admits a text-adapted predictor's residual correction to the base, and closing it exactly recovers the base model.Routing retains essentially all text-only gains on two held-out domains, leaves source-domain WER unchanged, and needs no target-domain audio tuning.Source: §6 · p. 4

8. COMPLIANCE WITH ETHICAL STANDARDS

The experiments used existing datasets and did not involve collecting new data from human participants or animals; therefore, no ethical approval was required.

8. COMPLIANCE WITH ETHICAL STANDARDS

  • Experiments used existing datasets, with no new human or animal data collection; ethical approval was therefore not required.Source: §8 · p. 5

Related papers

Loading 2610.32637v1…