PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Sequence Transduction by Jointly Predicting Tokens and Durations

Hainan Xu, Fei Jia, Somshubra Majumdar, He Huang, Shinji Watanabe, Boris Ginsburg

arXiv:2304.06795v2eess.AScs.CLcs.LGcs.SD

TL;DR

RNN-T models achieve strong speech-task accuracy, but autoregressive decoding is computationally costly. TDT jointly predicts tokens and durations to skip input frames during inference, and reports similar or better performance with faster inference across multiple tasks.

  • Problem

    RNN-T autoregressive decoding is computationally costly despite its strong accuracy in speech tasks.

    Source: §1 · p. 1

  • Method

    TDT extends Transducers by jointly predicting an output token and its duration, which guides frame skipping during inference.

    Source: §1 · p. 1, §3.4 · p. 4

  • Results

    Across speech recognition, speech translation, and spoken language understanding, TDT achieves better or similar performance while running inference faster, with up to 2.82X speed up.

    Source: §6 · p. 9

  • Takeaways & Limitations

    TDT is also reported to be more robust to noise and token repetition than conventional RNN-Ts.

    Source: §3 · p. 2, §6 · p. 9

    TDT beam search is highly complex because the search space spans both token and duration dimensions.

    Source: §4 · p. 5

Abstract

from arXiv · show

This paper introduces a novel Token-and-Duration Transducer (TDT) architecture for sequence-to-sequence tasks. TDT extends conventional RNN-Transducer architectures by jointly predicting both a token and its duration, i.e. the number of input frames covered by the emitted token. This is achieved by using a joint network with two outputs which are independently normalized to generate distributions over tokens and durations. During inference, TDT models can skip input frames guided by the predicted duration output, which makes them significantly faster than conventional Transducers which process the encoder output frame by frame. TDT models achieve both better accuracy and significantly faster inference than conventional Transducers on different sequence transduction tasks. TDT models for Speech Recognition achieve better accuracy and up to 2.82X faster inference than conventional Transducers. TDT models for Speech Translation achieve an absolute gain of over 1 BLEU on the MUST-C test compared with conventional Transducers, and its inference is 2.27X faster. In Speech Intent Classification and Slot Filling tasks, TDT models improve the intent accuracy by up to over 1% (absolute) over conventional Transducers, while running up to 1.28X faster. Our implementation of the TDT model will be open-sourced with the NeMo (https://github.com/NVIDIA/NeMo) toolkit.

1. Introduction

Transducer models are widely used, with research improving their encoders, decoders, and loss functions. This paper addresses their costly autoregressive inference with a Token-and-Duration Transducer and derives gradients for its training objective.

1. Introduction

  • End-to-end ASR includes attention-based encoder-decoder, CTC, and Transducer models, supported by open-source toolkits.The passage names ESPNet, NeMo, Espresso, and SpeechBrain.Source: §1 · p. 1
  • Prior Transducer research has modified encoders, decoders, and loss functions, including multi-blank symbols that cover multiple input frames.Examples include Transformer, Contextnet, and Conformer encoders, stateless and Echo State Network decoders, and FastEmit.Source: §1 · p. 1
  • TDT jointly predicts each output token and its duration, allowing predicted durations to guide frame skipping during inference.This targets the computational cost of autoregressive RNN-T decoding.Source: §1 · p. 1
  • The paper extends the forward-backward algorithm to derive analytical TDT gradients and derives token-logit gradients using Transducer function-merging.The function-merging approach is inspired by Li et al. (2019).Source: §1 · p. 1

3. TDT models achieve better accuracy and significant

The paper reports faster inference and robustness benefits for TDT models compared with conventional Transducers. Its alignment illustration shows duration-guided skipping concentrated in relatively silent audio segments.

3. TDT models achieve better accuracy and significant

  • In four relatively silent audio segments, the TDT alignment uses long durations to skip most frames, unlike the mostly horizontal conventional RNN-T alignment.Long skips are not frequently used where speech is present; each alignment time-axis unit corresponds to four spectrogram frames.Source: Figure 1 · p. 2
  • TDT inference speed-up is reported across speech recognition, speech translation, and spoken language understanding.The supplied passage does not give task-specific speed-up values.Source: §3 · p. 2
  • TDT models are reported to be more robust to noise and repeated-token speech than conventional RNN-Ts.Source: §3 · p. 2
  • The TDT implementation is planned for open release with NVIDIA’s NeMo toolkit.Source: §3 · p. 2

2. Background: Transducers

A conventional Transducer combines encoder, decoder, and joint-network representations to assign probabilities to output sequences. Its training objective sums probability across blank-augmented alignments, computed efficiently with forward or backward variables.

2. Background: Transducers

  • An RNN-Transducer has an encoder, decoder, and joint network that produces a vocabulary distribution from acoustic and text representations.The vocabulary includes a special blank symbol, and modern Transducers may use encoders beyond recurrent networks.Source: §2 · p. 2
  • Training maximizes the log probability of the target sequence by summing over ways to insert blanks into it.The blank-insertion operation B maps a target to augmented sequences, while B^-1 removes the blanks.Source: §2 · p. 2
  • Because direct summation over exponentially many augmented sequences is intractable, forward or backward variables compute the probability recursively.The recursion uses specified boundary conditions and domain constraints.Source: §2 · p. 2

3. Token-and-Duration Transducers

TDT adds explicit duration prediction to the Transducer joint network and incorporates duration choices into sequence-probability computation. The section derives training gradients and describes inference that advances through encoder frames by predicted durations.

3. Token-and-Duration Transducers

  • The TDT joint network emits token and duration distributions, which are independently normalized and jointly trained; token and duration are assumed conditionally independent.The duration distribution specifies the number of frames covered by an emission.Source: §3 · pp. 2–3, Figure 2 · p. 3
  • The forward recurrence sums over duration-weighted transitions, while the backward computation yields the whole-sequence probability.Blank emissions with duration zero are disallowed, so the model is not strictly probabilistic unless those duration probabilities are renormalized.Source: §3 · p. 3

3.1. TDT Gradient Computation

  • Analytical TDT-loss gradients are derived for token and duration probabilities because automatic differentiation for Transducer loss is highly inefficient.The section also describes a weighted sum of reachable backward variables used in the token-gradient computation.Source: §3.1 · pp. 3–4

3.2. Gradient with Transducer Function Merging

  • Transducer function-merging computes gradients directly with respect to pre-softmax token logits, but is not applied to duration logits.Duration logits have small dimensions, so the efficiency gain does not outweigh implementation complexity.Source: §3.2 · p. 4

3.3. Logits Under-normalization

  • Training-only logit under-normalization encourages emissions with longer durations; TDT inference instead increments the frame index by the predicted duration.Unlike conventional Transducer inference, which advances one frame only on blank emissions, TDT can advance by more than one frame.Source: §3.3 · p. 4, §3.4 · pp. 4–5

4. Experiments

Experiments evaluate TDT on speech recognition, speech translation, and spoken language understanding. Across these tasks, TDT models match or improve conventional Transducer accuracy while reducing inference time.

4. Experiments

  • The experiments cover speech recognition, speech translation, and spoken language understanding, using Conformer-Large models unless otherwise specified.The reported evaluations use non-batched greedy search and no external language model.Source: §4 · p. 5

4.1.1. ENGLISH ASR

4.1. Speech Recognition

  • ASR experiments use a conventional Transducer baseline and test consecutive duration sets, with configurations such as 0-4 denoting durations 0 through 4.Models use Conformer-Large encoders and stateless decoders; training uses public datasets for reproducibility.Source: §4.1 · p. 5

4.1.2. SPANISH ASR

  • TDT consistently improves Spanish Callhome WER over RNNT, with almost 2 absolute WER points for 0-2 and over 1 point for the fastest 0-8 model.Configurations 0-6 and 0-8 reach a maximum speed-up factor of 1.96.Source: Table 3 · p. 6, §4.1.2 · p. 6

4.1.3. GERMAN ASR

  • TDT has accuracy similar to or better than RNNT on German MLS and reaches 2.82X speed-up with configuration 0-8.Source: Table 4 · p. 6, §4.1.3 · p. 6

4.2. Speech Translation

  • TDT improves MUST-C V2 Test BLEU by up to 1.26 points over RNNT and runs up to 2.27X faster.The paper notes that its RNNT and TDT results are not directly comparable with the publicly available reference model because they use different training data.Source: Table 5 · p. 6, §4.2 · p. 6

4.3. Spoken Language Understanding

  • TDT [0-4], [0-6], and [0-8] improve SICSF accuracy to a new state of the art, while [0-8] runs 1.28X faster than RNNT.SLURP has longer text than audio on average, which the paper associates with smaller speedups than in ASR and speech translation. The RNNT baseline was initialized from pretrained ASR; comparison systems used HuBERT encoders.Source: Table 6 · p. 7, §4.3 · p. 7

5. Discussion

TDT’s duration predictions reduce decoding work, and the discussion reports accuracy and speed benefits across batched inference, noisy speech, repeated-token speech, and comparisons with multi-blank Transducers. Emission analyses show how longer predicted durations change decoding behavior.

5.1. TDT Emission Analysis

  • On Librispeech test-other, longer-duration configurations produce fewer blank emissions while non-blank emission counts remain unchanged.Durations [0–2] and [0–4] are predicted at their maximum for nearly all and around 90% of outputs, respectively; the average audio-to-text length ratio is 5.5:1, limiting duration 6.Source: §5.1 · p. 7, Figure 5 · p. 8

5.2. TDT Batched Inference

  • Batched TDT inference with loss sampling shows slightly improved ASR accuracy and inference speedup, despite smaller speedups than non-batched inference.Padding overhead and advancing all utterances by the batch’s minimum predicted duration increase decoding steps.Source: §5.2 · p. 8, Table 7 · p. 8

5.3. TDT Robustness to noise

  • TDT models gradually outperform RNN-Ts in WER as noise increases, while their inference time remains practically the same across SNRs.On clean speech, the models achieve similar WERs.Source: Figure 6 · p. 8, §5.3 · p. 8

5.4. TDT Robustness with respect to repeated tokens

  • 5.78% WER with durations 0–8 was more than a 10X error-rate reduction over conventional Transducers on repeated-digit speech.Conventional RNN-Ts had error rates above 50%, regardless of decoder type.Source: §5.4 · p. 9, Table 8 · p. 9

5.5. TDT Comparison with Multi-blank Transducers

  • TDT and multi-blank Transducers give comparable WERs, but TDT has larger inference speedups at the same maximum-duration configurations.TDT can skip frames for both blank and non-blank symbols, whereas multi-blank Transducers skip only with certain blank symbols.Source: §5.5 · p. 9, Table 9 · p. 9

6. Conclusion

The conclusion presents TDT as an extension of conventional Transducers with explicit duration modeling. Across speech recognition, speech translation, and spoken language understanding, it reports similar or better performance with faster inference.

6. Conclusion

  • Across speech recognition, speech translation, and spoken language understanding, TDT achieves better or similar performance and runs inference significantly faster, with speedups up to 2.82X.The implementation is planned for open release in NVIDIA’s NeMo toolkit.Source: §6 · p. 9
  • Future work will explore computational efficiency and accuracy improvements, as well as efficient TDT beam-search algorithms and implementations.Source: §6 · p. 9

A. Derivations of TDT gradients with respect to probabilities

The appendix derives probability and logit gradients for conventional Transducers and TDT, accounting for the frame-skipping transitions enabled by TDT. It also outlines the softmax-function-merging gradient calculations.

A.1. Background: gradients of RNN-Transducer

  • The conventional Transducer probability derivation uses forward-backward terms and differentiates sequence probability with respect to individual output probabilities.The appendix expresses sequence probability as a sum over a lattice diagonal and substitutes the backward recurrence.Source: §A.1 · p. 12

A.2. Gradients of TDT models

  • TDT gradient derivations account for paths that jump over lattice diagonals, then derive gradients for token and duration probabilities.Frame-skipping means the conventional Transducer diagonal property does not hold for TDT.Source: §A.2 · pp. 12–13

B.1. Background: Softmax function merging for conventional Transducers

  • In the conventional Transducer logit-gradient derivation, only blank and next-token probabilities contribute nonzero terms to the probability-gradient sum.Source: §B.1 · p. 13

B.2. Softmax function merging for TDT

  • The TDT pre-softmax logit-gradient derivation applies the chain rule and simplifies the resulting terms according to the predicted symbol.Source: §B.2 · p. 14

B.2.2. THE CASE WHEN v = yu+1

  • For v = yu+1, the TDT logit-gradient case combines the preceding equation results.Source: §B.2.2 · p. 16

C. Derivation of TDT Gradients with Logit Under-normalization

The appendix derives TDT gradients when token logits use under-normalization, which is applied to token logits but not duration logits. It also illustrates how alignments vary with duration-token count and FastEmit strength.

C. Derivation of TDT Gradients with Logit Under-normalization

C.3. When v = yu+1

  • With more duration tokens, simulated-joint alignments show longer gaps between token predictions; increasing FastEmit strength corresponds to shorter delays.Source: Figure 7 · p. 19, Figure 8 · p. 19

D. Analysis of Token-and-Duration Transducer Alignments

The alignment experiments examine how duration choices, FastEmit, and the input-to-target length ratio shape TDT token timing. Longer durations reduce decoding steps but can delay emissions; sequence-length effects and FastEmit can alter that trade-off.

D. Analysis of Token-and-Duration Transducer Alignments

  • The alignment analysis defines P(At,u|x) as the total probability mass passing through state (t, u) in the lattice and computes it for simulated-joint experiments.The default setup optimizes the simulated joint tensor with the TDT loss and plots the resulting T × U alignment matrix.Source: D. Analysis of Token-and-Duration… · p. 19

D.1. Effect of Durations on TDT Alignments

  • As the duration set grows, TDT selects longer durations, reducing decoding steps while significantly delaying token emissions relative to the single-step baseline.The experiments use a simulated joint tensor with T = 70 and U = 10; longer durations are enabled by this 7:1 ratio.Source: §D.1 · pp. 19–20

D.2. Effect of FastEmit on Alignments

  • FastEmit substantially reduces token-emission delay, but with Nd = 8, emissions remain significantly delayed because each token spans roughly eight timesteps.The first 5–6 emissions are delayed for Nd = 8 compared with Nd = 1; the experiments also test λ values that would cause divergence on non-synthetic data when λ > 1e-2.Source: §D.2 · p. 20

D.3. Effect of Input Sequence length on Duration Prediction

  • When T/U is small, short durations are preferred and emissions have little delay; when T/U is large, long durations, including d = 8, are preferred and emissions are delayed.Increasing FastEmit strength reduces token latency even at large input-to-target length differences.Source: §D.3 · p. 20

E. Robustness to Noise

The noise evaluation compares conventional Transducers with TDT models on noisy Librispeech test-clean audio. TDT maintains similar WER in high-SNR conditions, becomes more robust as noise increases, and has only minimal inference-time increases as SNR changes.

E. Robustness to Noise

  • As noise increases, TDT models increasingly outperform conventional Transducers, while their inference time increases only minimally across SNRs.The comparison uses Librispeech test-clean augmented at SNRs of 0, 5, 10, 15, and 20, with TDT configuration 0-8.Source: E. Robustness to Noise · pp. 20–21, Figure 11 · p. 22
  • As the input-to-target length ratio grows, alignments favor longer durations and increase token-emission delay; increasing FastEmit strength reduces that delay.The alignment figures describe this relationship for simulated joints, including experiments with different input lengths.Source: Figure 9 · p. 21, E. Robustness to Noise · p. 21, Figure 10 · p. 21
  • At low noise, TDT and RNN-T achieve similar WER; at low SNR, TDT is more robust.The original test-clean data is listed at SNR = +inf.Source: E. Robustness to Noise · p. 20, Figure 11 · p. 22

F. Robustness to Token Repetitions

The section describes repeated-token decoding as a source of RNN-T performance degradation and compares models on repeated-digit speech. TDT is less prone to repetition issues, and the reported evaluation finds it more robust than RNN-T on that dataset.

F. Robustness to Token Repetitions

  • RNN-T non-blank emissions do not advance the input frame, so repeated tokens can keep decoding on the same encoder frame and produce insertion errors.The paper notes that this issue can occur with just two repetitions, not only long runs.Source: F. Robustness to Token Repetitions · pp. 22–23
  • TDT models are much more robust than RNN-Ts on a TTS-generated repeated-digit speech dataset.The evaluation uses 100 utterances, each containing three randomly chosen digits repeated 3–5 times; the available table caption identifies the comparison but supplies no values here.Source: F. Robustness to Token Repetitions · p. 23, Table 10 · p. 23
  • TDT is less prone to repetition issues because its duration output makes it less likely to remain on the same frame across decoding steps.The paper reports that duration 0 is rarely emitted.Source: F. Robustness to Token Repetitions · p. 23
Loading 2304.06795v2…