PaperlayerView exact revision on arXiv

Source-linked AI summary

MAM: Masked Acoustic Modeling for End-to-End Speech-to-Text Translation

Junkun Chen, Mingbo Ma, Renjie Zheng, Liang Huang

arXiv:2010.11445v2cs.CL

TL;DR

Existing approaches to improve end-to-end speech translation rely on source-language transcriptions, which can be costly or unavailable. MAM learns by recovering masked speech from context without transcriptions; across 8 translation directions, it improves average BLEU by 1.09 in the basic setting and 2.26 with pre-training.

  • Problem

    Approaches that improve end-to-end speech translation rely on source-language transcriptions, which are expensive to collect and may be impossible for languages without a written form or standard orthography.

    Source: §1 · p. 1

  • Method

    MAM randomly masks portions of speech and trains an encoder to recover the masked signals from context, without using source-language transcriptions.

    Source: §1 · p. 2

  • Results

    Across 8 translation directions without transcriptions, MAM improves average BLEU by 1.09 in the basic setting and 2.26 with pre-training.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MAM pre-training with arbitrary acoustic data boosts speech translation performance.

    Source: §6 · p. 9

    MAM trained with Common Voice does not significantly improve results; its speech clips can be only 2 to 3 seconds long, limiting random masking options, and the corpus contains much less English speech.

    Source: §4.3.2 · p. 7

Abstract

from arXiv · show

End-to-end Speech-to-text Translation (E2E-ST), which directly translates source language speech to target language text, is widely useful in practice, but traditional cascaded approaches (ASR+MT) often suffer from error propagation in the pipeline. On the other hand, existing end-to-end solutions heavily depend on the source language transcriptions for pre-training or multi-task training with Automatic Speech Recognition (ASR). We instead propose a simple technique to learn a robust speech encoder in a self-supervised fashion only on the speech side, which can utilize speech data without transcription. This technique termed Masked Acoustic Modeling (MAM), not only provides an alternative solution to improving E2E-ST, but also can perform pre-training on any acoustic signals (including non-speech ones) without annotation. We conduct our experiments over 8 different translation directions. In the setting without using any transcriptions, our technique achieves an average improvement of +1.1 BLEU, and +2.3 BLEU with MAM pre-training. Pre-training of MAM with arbitrary acoustic signals also has an average improvement with +1.6 BLEU for those languages. Compared with ASR multi-task learning solution, which replies on transcription during training, our pre-trained MAM model, which does not use transcription, achieves similar accuracy.

1 Introduction

Cascaded speech translation risks error propagation, while methods that improve end-to-end speech translation rely on source-language transcriptions. MAM learns from masked speech without transcriptions and can also pre-train on arbitrary acoustic signals.

1 Introduction

  • Cascaded speech translation first transcribes speech and then translates it, a pipeline that inevitably suffers from error propagation.End-to-end approaches mitigate this risk, while existing encoder pre-training and ASR multi-task methods rely on source-language transcriptions.Source: §1 · p. 1
  • Source-language transcriptions are expensive to collect, and some languages lack a written form or standard orthography, making phonetic transcription impossible.The paper notes that far more speech audio is available without annotations than with them.Source: §1 · pp. 1–2
  • MAM randomly masks portions of speech and trains the encoder to recover them from context, without source-language transcriptions.The paper also presents MAM as a pre-training module for arbitrary acoustic signals, including music and animal sounds.Source: §1 · p. 2
  • Across 8 translation directions, MAM improves average BLEU by 1.09 in the basic setting and 2.26 with pre-training when no transcriptions are used.MAM adds 6.5% more parameters than the baseline model.Source: §1 · p. 2

2 Preliminaries: ASR and ST

The section introduces the sequence-to-sequence setup for end-to-end speech translation and then describes how ASR multi-task learning shares its encoder and adds a transcription objective.

2 Preliminaries: ASR and ST

  • The preliminaries review standard end-to-end speech translation and its ASR multi-task variant to establish notation.Source: §2 · p. 2

2.1 Vanilla E2E-ST Training with Seq2Seq

  • End-to-end speech translation uses an encoder-decoder sequence-to-sequence model, with the decoder predicting target words from the source and its previously decoded prefix.The encoder maps source speech features to hidden representations; training maximizes the conditional probability of the ground-truth target sentence.Source: §2.1 · p. 2

2.2 Multi-task Learning with ASR

  • ASR multi-task learning shares an encoder between translation and transcription decoders, adding an ASR objective to ST training.The section presents ASR training as helping the encoder learn speech segmentations that make global reordering for translation easier.Source: §2.2 · p. 2

3 Masked Acoustic Modeling

MAM adds a self-supervised encoder objective that reconstructs masked speech from context, removing the need for source-language transcripts. Its masking also supports encoder pretraining on arbitrary audio, including signals beyond speech.

3 Masked Acoustic Modeling

  • Existing E2E-ST improvements rely on source-language transcripts, limiting use of unannotated speech and languages without standard orthography.Source: §3 · pp. 2–3

3.1 MAM as Part of Training Objective

  • MAM randomly masks speech frames and trains the encoder to reconstruct the original signal from corrupted-input representations and surrounding context.The reconstruction function uses 2D deconvolution and mean squared error.Source: §3.1 · p. 3

3.2 Different Masking Strategies

  • The paper varies masking granularity with uniform single-frame masking or non-overlapping, randomly placed spans.Source: §3.2 · p. 3

3.3 Pre-training MAM

  • Pretraining uses only the encoder and supports source-language, multilingual, or arbitrary audio without annotation; arbitrary-audio pretraining need not include the downstream source language.Source: §3.3 · pp. 3–4

4 Experiments

Experiments use MuST-C across eight English-source translation directions and evaluate MAM pretraining with several audio corpora. MAM adds fewer parameters than ASR multi-tasking, and the paper reports that the design is computationally modest to train.

4 Experiments

  • Pretraining uses Libri-Light, 14-language Common Voice, and Audioset, whose arbitrary sounds include human, animal, environmental, and musical audio.Common Voice has transcripts, but the experiments do not use them.Source: §4 · p. 4, Table 2 · p. 4
  • The evaluation covers eight English-source translation directions on the MuST-C dataset.Source: §4 · p. 4
  • MAM adds 6.5% more parameters than the baseline, while ASR multi-tasking needs 51.6% more parameters.Source: Table 1 · p. 4
  • The baseline uses a 12-layer Transformer with 2048 hidden size; MAM projects encoder outputs to a latent space and upsamples them with two deconvolution layers.The masking ratio is 30% across experiments, and inference uses no masking.Source: §4 · p. 4
  • The basic MAM model converges within two days using eight 1080Ti GPUs, without parameter or architecture search.Source: §4 · p. 4

4.1 Analyzing ASR and MAM

The section analyzes encoder self-attention to explain why ASR and MAM help E2E-ST. MAM produces attention patterns resembling ASR, including speech chunking, and can serve as an alternative when transcriptions are unavailable.

4.1 Analyzing ASR and MAM

  • E2E-ST must map speech to text across modalities, handle speech inputs 4 to 7 times longer than target sequences, and learn global reordering across languages.The paper also notes the challenge of finding corresponding phoneme or syllable segments from a different-language training signal.Source: §4.1 · pp. 4–5
  • ASR MTL encourages meaningful encoder self-attention that segments spectrogram frames into phoneme- or syllable-level groups, reducing decoder alignment to reordering segments.The monotonicity of ASR alignment functions as a forced alignment for adjacent frames.Source: §4.1 · p. 5
  • MAM without pre-training also segments source speech into chunks, while pre-training alone yields mostly monotonic attention that can focus on longer-distance frames.Fine-tuning on ASR or ST produces chunked top-layer attention similar to the patterns seen with ASR MTL and MAM during E2E-ST training.Source: §4.1 · p. 5
  • MAM functions similarly to ASR on the encoder side, offering an alternative when transcriptions are unavailable.After MAM pre-training, encoder self-attention is mostly monotonic; downstream fine-tuning produces chunked attention.Source: §4.1 · p. 5

4.2 Visualizing Reconstruction

The section visualizes how pre-trained MAM reconstructs corrupted spectrograms using speech and music data. The authors report similar reconstruction for corrupted speech after pre-training on either corpus.

4.2 Visualizing Reconstruction

  • The FMA medium setting contains about 25,000 30-second music tracks across 16 unbalanced genres, totaling about 208 hours.The authors choose FMA because music differs substantially from pure human speech.Source: §4.2 · p. 6
  • The reconstructed spectrograms are somewhat blurrier than the original inputs, which the authors attribute to downsampling in the E2E-ST baseline framework.They suggest FMA reconstruction may reflect learning from tracks that include human singing voices.Source: §4.2 · p. 6
  • The visualization compares original spectrograms with inputs corrupted by replacing selected mask frames with a random initialized vector.The original speech example includes a transcription annotation, but transcription is not used during pre-training.Source: §4.2 · p. 6
  • MAM recovers masked speech segments after pre-training on Libri-Light, and FMA pre-training yields similar reconstruction of corrupted speech.Figure 4 presents an example of reconstruction from ordinary speech and music pre-training.Source: §4.2 · p. 6, Figure 4 · p. 6

4.3 Translation Accuracy Comparisons

MAM improves E2E-ST across comparison settings, including without transcription, with speech pre-training, and alongside ASR multitask training; its arbitrary-audio pre-training also outperforms the baseline.

4.3 Translation Accuracy Comparisons

  • The six comparison baselines include cascaded ASR+MT, transcription-based MT, vanilla E2E-ST, ASR multitask learning, SpecAugment, and ASR-pretrained E2E-ST.The section frames its translation-accuracy comparisons against six baselines.Source: §4.3 · p. 6

4.3.1 Comparison in the settings without transcription and pre-training

  • +1.09 average BLEU over E2E-ST came from span-masked MAM without pre-training, exceeding the +0.44 average for single masking across eight translation directions.The authors attribute span masking's consistent advantage to its being a harder self-supervised task.Source: §4.3.1 · p. 6, Table 3 · p. 7

4.3.2 Comparison in the pre-training settings without transcription

  • +2.26 BLEU over E2E-ST and about +0.79 BLEU over E2E-ST + ASR PT were reported for Libri-Light MAM pre-training across eight target languages.The Table 4 caption notes that MAM methods do not use transcription and that English-corpus pre-training performs similarly to E2E-ST + ASR MTL.Source: §4.3.2 · p. 7, Table 4 · p. 7
  • About +1.55 improvement over E2E-ST was reported when MAM was pre-trained on arbitrary acoustic data, which requires no annotation.The experiments also included English speech and multilingual speech pre-training; Common Voice showed no significant improvement, with short clips and fewer English examples cited as reasons.Source: §4.3.2 · pp. 6–7

4.3.3 Comparison in the settings using transcription

  • +0.9 average improvement over E2E-ST + ASR MTL was reported for MAM multitask learning with Libri-Light pre-training across eight languages.This setting uses ASR MTL as its baseline.Source: §4.3.3 · p. 7

4.3.4 Comparison to Wav2vec

  • MAM achieved +1.6 more BLEU improvement than wav2vec-based pre-training methods, with the paper reporting an approximately 5.6-BLEU advantage on En-Ro.The comparison uses relative improvements on the same test data because the baseline methods differ; the paper says its own baseline was already about four BLEU points better than the wav2vec baseline.Source: §4.3.4 · p. 7

4.4 Comparisons in Low and Mid-resource Settings

In reduced-resource comparisons, MAM pre-training retains similar performance in specified settings without transcription, and its reported gains exceed those of wav2vec-based pre-training methods.

4.4 Comparisons in Low and Mid-resource Settings

  • MAM's relative improvements over wav2vec-based pre-training methods are reported as much larger, even against a stronger baseline.The passage refers readers to the wav2vec comparison for further discussion.Source: §4.4 · p. 8
  • The section reports that multilingual and acoustic pre-training boost performance significantly.The table caption describes comparisons across different training-resource settings.Source: §4.4 · p. 8, Table 7 · p. 8
  • When transcription is available only for fine-tuning, Libri-Light MAM pre-training achieves similar performance without using transcription during fine-tuning.The stated comparison is against “+ASR MTL.”Source: §4.4 · p. 8
  • When fine-tuning uses no transcription but large-scale annotated pre-training data is available, MAM still achieves similar performance in the 200-hour training setting without using any transcription.The experiment reduces MuST-C from 408 hours to 50 and 200 hours to mimic low- and mid-resource language speech translation.Source: §4.4 · pp. 7–8

5 Related Work

Related work includes text-masking, speech augmentation, forced-alignment pre-training, and wav2vec approaches; MAM differs in reconstructing speech signals without transcription-dependent alignment.

5 Related Work

  • BERT-style methods recover missing semantic information, whereas MAM also restores speech characteristics and generates the original speech.The cited text-based frameworks include Devlin et al. (2019), Liu et al. (2019a), Joshi et al. (2020a), and Zhang et al. (2019).Source: §5 · p. 8
  • SpecAugment masks input speech for augmentation but has no recovery step and cannot serve as a pre-training framework.The passage cites Park et al. (2019) and Bahar et al. (2019).Source: §5 · p. 8
  • Forced-alignment speech pre-training masks word-level segments and depends on source transcriptions, limiting its use for low- or zero-resource speech.The passage cites Chuang et al. (2019), Wang et al. (2020c;a), Povey et al. (2011), and McAuliffe et al. (2017); it contrasts this with MAM's reduced need for annotated speech and translation pairs during pre-training.Source: §5 · p. 8
  • Wav2vec 2.0 masks latent speech representations and uses a quantized contrastive objective, while its CTC-based ASR fine-tuning is not straightforward for translation.The passage also notes that its quantized codebooks are not easy to adapt to arbitrary acoustic signals.Source: §5 · p. 8

6 Conclusions

The paper concludes that MAM can serve both as an additional training component and as a separate pre-training framework, including for arbitrary acoustic signals.

6 Conclusions

  • MAM is presented as both an extra component during training and a separate pre-training framework using arbitrary acoustic signals.The conclusion reports experiments across eight languages and highlights arbitrary-acoustic-data pre-training as a first demonstration for speech translation.Source: §6 · pp. 8–9

Appendix

Libri-Light-pretrained MAM was tested on corrupted music, but it did not reconstruct the spectrogram accurately because musical-instrument sounds were unseen in human speech data.

Appendix

  • Libri-Light-pretrained MAM did not reconstruct corrupted music accurately because many instrument sounds were unseen in its human-speech training data.The example used an original musical spectrogram mixed from different instruments.Source: Appendix · p. 12, Figure A1 · p. 12
  • The reconstruction test applied speech-pretrained MAM to corrupted music.The original spectrogram contained sounds from different instruments.Source: Appendix · p. 12
  • Figure A1 presents an example of MAM's music-spectrogram reconstruction ability.The example highlights poor reconstruction when the model was pretrained on human speech only.Source: Figure A1 · p. 12, Appendix · p. 12
Loading 2010.11445v2…