PaperlayerView exact revision on arXiv

Source-linked AI summary

Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translation

Shun-Po Chuang, Yung-Sung Chuang, Chih-Chiang Chang, Hung-yi Lee

arXiv:2105.04840v1cs.CL

TL;DR

Autoregressive decoding creates latency for end-to-end speech translation, while non-autoregressive approaches are rarely explored in speech translation. This paper uses CTC with a transformer encoder and auxiliary CTC-based ASR, then analyzes reordering; its analysis finds non-autoregressive models can rearrange words, though less strongly than autoregressive models.

  • Problem

    Autoregressive decoding causes latency in end-to-end speech translation, while non-autoregressive models are rarely explored for speech translation.

    Source: §1 · p. 1

  • Method

    The paper trains non-autoregressive speech translation with CTC on a transformer encoder, adds auxiliary CTC-based ASR, and analyzes reordering using Kendall’s tau distance and gradient visualization.

    Source: §1 · p. 1, §2.3 · p. 2

  • Results

    Non-autoregressive models can rearrange words but are weaker than autoregressive models on Racc; applying multitask learning tends to improve Racc.

    Source: §3.3 · p. 4

  • Takeaways & Limitations

    Non-autoregressive speech translation research should evaluate reordering capability as well as BLEU to investigate the gap with autoregressive models.

    Source: §4 · p. 4

    CTC assumes independent token generation and monotonic alignment, although translation word orders can differ across languages.

    Source: §2.1 · p. 2

Abstract

from arXiv · show

We study the possibilities of building a non-autoregressive speech-to-text translation model using connectionist temporal classification (CTC), and use CTC-based automatic speech recognition as an auxiliary task to improve the performance. CTC's success on translation is counter-intuitive due to its monotonicity assumption, so we analyze its reordering capability. Kendall's tau distance is introduced as the quantitative metric, and gradient-based visualization provides an intuitive way to take a closer look into the model. Our analysis shows that transformer encoders have the ability to change the word order and points out the future research direction that worth being explored more on non-autoregressive speech translation.

1 Introduction

End-to-end speech translation can avoid the ASR error bottleneck and improve computational efficiency, but autoregressive decoding introduces latency. This work tests CTC-based non-autoregressive translation with a transformer encoder and analyzes its reordering capability.

1 Introduction

  • Autoregressive decoding causes latency in end-to-end speech translation, despite its potential to avoid the ASR error bottleneck and improve computational efficiency.The passage contrasts end-to-end models with cascaded ASR and MT systems.Source: §1 · p. 1
  • The introduction situates non-autoregressive models as heavily explored in machine translation but rarely explored in speech translation, apart from a concurrent work.The supplied passage names Gu et al. (2018), Lee et al. (2018), Ghazvininejad et al. (2019), Stern et al. (2019), Gu et al. (2019), Saharia et al. (2020), and Vaswani et al. (2017) in describing the MT context.Source: §1 · p. 1
  • The authors use CTC to train non-autoregressive speech translation with a transformer encoder and no explicit decoder, adding CTC-based ASR multitask learning to improve performance.They report initial results for non-autoregressive translation from speech using a single speech encoder.Source: §1 · p. 1
  • Because CTC assumes monotonicity, the authors analyze reordering with Kendall’s tau distance and gradient-based visualization, describing this as the first such examination for speech translation.Earlier works applied CTC loss to non-autoregressive translation without verifying its reordering capability (Libovick`y and Helcl, 2018; Saharia et al., 2020; Inaguma et al., 2020).Source: §1 · p. 1

2 Approaches

The approach uses a transformer encoder trained with CTC for one-step non-autoregressive translation, adds CTC-based ASR at one encoder layer, and evaluates reordering through alignment disagreement.

2.1 CTC-based NAR-ST Model

  • CTC trains a convolutional and self-attention encoder by marginalizing latent alignments, enabling single-step translation from downsampled acoustic features.Blank-containing alignments are collapsed to the target sequence.Source: §2.1 · p. 2

2.2 CTC-based Multitask NAR-ST Model

  • The model applies CTC-based ASR at one encoder layer, while ASR output is excluded from translation decoding.This adds an auxiliary task without an ASR decoder in the translation decoding process.Source: §2.2 · p. 2

2.3 Reordering Evaluation – Kendall’s Tau Distance

  • Reordering correctness compares source-to-reference and source-to-hypothesis alignments by their proportion of disagreements, with a brevity penalty.The alignments map source tokens to reference and hypothesis tokens, respectively; higher correctness indicates greater alignment similarity.Source: §2.3 · pp. 2–3

3 Experiments

The experiments evaluate CTC-based non-autoregressive speech translation for translation speed and quality, then examine its word-reordering ability using Kendall’s tau distance and gradient-based visualizations.

3.1 Experimental Setup

  • Experiments use Spanish-to-English Fisher speech, include CALLHOME test sets, and evaluate a model with two convolutional and twelve transformer encoder layers.Knowledge distillation is also applied.Source: §3.1 · p. 3

3.2 Translation Quality and Speed

  • For AR models, MTL improves performance over training without an auxiliary task, while reducing beam size trades performance for decoding speed.The authors also report further improvement from initializing with a pretrained ASR encoder.Source: §3.2 · p. 3
  • NAR-ST decodes ×28.9 faster than AR with beam-size 10 and ×3.4 faster than AR with greedy decoding; applying MTL at higher layers improves performance.The experiments also indicate that choosing the intermediate layer for MTL matters.Source: §3.2 · p. 3

3.3 Word Order Analysis

  • AR models achieve high Racc; NAR models can reorder words but are weaker, and applying MTL tends to improve NAR reordering correctness.Table 2 reports reordering correctness on Fisher Spanish, with values selected as the best among four references.Source: §3.3 · p. 4, Table 2 · p. 4
  • AR models are more robust as reordering difficulty rises, while layer-8 MTL makes CTC more robust and sometimes brings it close to AR without ASR pretraining for Rπ<0.07.Figure 2 plots BLEU across groups with different reordering difficulties.Source: §3.3 · p. 4, Figure 2 · p. 4
  • For the token “thing,” MTL shifts attention across positions in lower layers toward the corresponding speech segment, while single-task CTC tends to focus on the same position.Figure 3 shows the reordering matrix and includes MTL ASR output as a proxy for source-speech word position.Source: §3.3 · p. 4, Figure 3 · p. 4

4 Concluding Remarks

The concluding remarks present CTC-based NAR speech translation with auxiliary CTC-based ASR and examine its reordering behavior. They encourage future NAR research to evaluate reordering capability as well as BLEU.

4 Concluding Remarks

  • The work proposes a CTC-based NAR-ST model.Source: §4 · p. 4
  • The paper proposes a CTC-based NAR speech translation model with an auxiliary CTC-based ASR task and studies its reordering capability.It uses Racc and gradient-based visualizations to analyze reordering.Source: §4 · p. 4
  • The authors encourage future NAR research to evaluate whether models have inferior reordering capability, in addition to measuring BLEU.Source: §4 · p. 4

Broader Impact and Ethical Considerations

The authors expect the work to help NLP researchers understand NAR speech translation and envision the proposed model as a technique for faster, better speech translation. They provide code and reproduction instructions.

Broader Impact and Ethical Considerations

  • The authors envision the proposed model as a technique for better and faster speech translation and report no perceived conflict with the ACL-IJCNLP 2021 code of ethics.Source: Broader Impact and Ethical… · p. 5

A Source Code

  • The authors provide a code repository and direct readers to its README for instructions to reproduce the results.Source: A Source Code · p. 7

B Dataset Details

The experiments use Spanish-to-English speech translation data from Fisher and CALLHOME, including speech corpora and a speech-translation dataset.

B Dataset Details

B.1 Statistics

The appendix points to a table of Fisher and CALLHOME Spanish data statistics and describes the text and audio preprocessing used.

B.1 Statistics

B.2 Preprocessing

  • Text is lowercased, stripped of punctuation, and encoded with 8,000-unit BPE.BPE is cited to Sennrich et al. (2016).Source: §B.2 · p. 7
  • Audio is converted to 16 kHz WAV, represented by 80-dimensional fbank features without deltas, and augmented with SpecAugment.Preprocessing uses ESPnet; SpecAugment is cited to Park et al. (2019).Source: §B.2 · p. 7

C Training Details

Experiments used a single NVIDIA TITAN RTX (24G), and the reported average runtime was 2–3 days for both AR and NAR models.

C.1 Computing Infrastructure and Runtime

  • Experiments ran on a single NVIDIA TITAN RTX with 24G of memory.Source: §C.1 · p. 7
  • Average runtime was 2–3 days for both autoregressive and non-autoregressive models.Source: §C.1 · p. 7

C.2 Hyperparameters

The experiments followed ESPnet's best autoregressive speech-translation setting without hyperparameter search, using the same hyperparameters for non-autoregressive models.

C.2 Hyperparameters

  • The experiments used ESPnet's best autoregressive speech-translation setting and applied the same hyperparameters to non-autoregressive models.No hyperparameter search was conducted.Source: §C.2 · p. 7
  • Each experiment was run once because of the limited budget.Source: §C.2 · p. 7
  • CTC-based models used greedy decoding at inference.Source: §C.2 · p. 7

C.3 Knowledge Distillation

The appendix describes sequence-level knowledge distillation, reports that MT-generated hypotheses performed better than ST-generated ones, and specifies model selection and parameter counts.

C.3 Knowledge Distillation

  • Seq-KD used an autoregressive transformer MT model trained on the training-set transcriptions and translations to generate beam-size-1 hypotheses, which replaced NAR models' ground-truth sequences.The ablation used CTC+MTL at the eighth layer on Fisher Spanish and CALLHOME.Source: §C.3 · p. 7, Table 5 · p. 8
  • Using autoregressive ST hypotheses for Seq-KD produced worse results than using MT hypotheses.Source: §C.3 · p. 7

C.4 Model Selection

  • NAR models were averaged over the final 10 epoch checkpoints, whereas AR models used the five best-validated checkpoints.Source: §C.4 · p. 8

C.5 Model Size

  • The CTC model has 18.2M parameters, compared with 27.9M for the autoregressive model.Source: §C.5 · p. 8

D Automatic Speech Recognition Evaluation

This section evaluates ASR-branch behavior and develops alignment-based analyses of reordering, finding that reordering correctness falls with difficulty similarly for NAR and AR models.

D Automatic Speech Recognition Evaluation

  • Higher-layer MTL lowers ASR WER, but CTC+MTL@8—not @10—achieves the best ST scores; the authors suggest @10 leaves only two encoder layers for translation.The lower WER is described as providing useful information for ST, despite the possible translation difficulty at @10.Source: D Automatic Speech Recognition… · p. 8, Table 6 · p. 8

E SimAlign setup

  • SimAlign obtains word alignments using XLM-R contextualized embeddings and the itermax matching algorithm.The alignments are used to calculate reordering metrics.Source: E SimAlign setup · p. 8

F Reordering Difficulty

G Details on Figure 2

  • Figure 2 measures difficulty by comparing reference alignments with a dummy monotonic alignment, then plots BLEU across ten equally sized difficulty bins.The bins contain examples with similar reordering difficulty.Source: G Details on Figure 2 · p. 8

H Gradient-based Visualization

  • Normalized gradients over audio positions indicate each position’s relative influence on an output token; concatenating these influences across layers forms its reordering matrix.Source: H Gradient-based Visualization · p. 9

I Reordering Matrix

J Higher Reordering Difficulty

  • Reordering correctness decreases consistently as difficulty rises, at a similar rate for NAR and AR models; the authors expect their gap to remain roughly similar on distant language pairs.They plan future experiments on different language pairs to verify this expectation.Source: J Higher Reordering Difficulty · p. 9, Figure 7 · p. 10
Loading 2105.04840v1…