Source-linked AI summary
End-to-End Automatic Speech Translation of Audiobooks
Alexandre Bérard, Laurent Besacier, Ali Can Kocabiyikoglu, Olivier Pietquin
TL;DR
The paper addresses end-to-end speech translation for audiobooks, studying whether source transcripts are needed during learning and decoding. It augments LibriSpeech, trains single-pass models with training-time transcripts in an intermediate setup, and finds that transcript-informed end-to-end methods approach cascaded performance while enabling compact models.
Problem
Large publicly available speech-to-text translation corpora are scarce, and prior end-to-end work mainly studied settings without source transcription during learning or decoding.
Method
The authors augment LibriSpeech with English speech, transcripts, and French translations, then compare direct end-to-end AST with cascaded ASR and MT models.
Results
Cascading ASR and MT gives the best results, while end-to-end methods incorporating source transcripts come close in performance.
Takeaways & Limitations
The corpus provides audiobook speech-translation baselines, and compact end-to-end models are feasible when source transcripts are available during training.
Takeaways & Limitations
The intermediate setup assumes source transcriptions are available during training, but not during decoding.
Abstract
from arXiv · showhide
We investigate end-to-end speech-to-text translation on a corpus of audiobooks specifically augmented for this task. Previous works investigated the extreme case where source language transcription is not available during learning nor decoding, but we also study a midway case where source language transcription is available at training time only. In this case, a single model is trained to decode source speech into target text in a single pass. Experimental results show that it is possible to train compact and efficient end-to-end speech translation models in this setup. We also distribute the corpus and hope that our speech translation baseline on this corpus will be challenged in the future.
1. INTRODUCTION
The paper studies end-to-end speech-to-text translation for audiobooks, extending prior work beyond the case without source transcription. It focuses on an intermediate setup where transcripts are available during training but not decoding.
- Motivation: Most speech translation systems combine source-language ASR with source-to-target MT through an intermediate symbolic sequence.Recent work instead attempts direct speech-to-text translation without source transcription during learning or decoding.
- 研究対象: The study applies end-to-end speech translation to LibriSpeech, an audiobook corpus augmented for this task.The paper follows previous work while evaluating on a dedicated audiobook corpus.
- Intermediate scenario: The intermediate setup makes source transcription available during training but uses one end-to-end model to decode speech into target text in a single pass.This setup is intended for compact speech translation models.
- Evaluation: Experiments cover both a synthetic BTEC dataset and the augmented audiobook dataset.The paper presents the corpus, models, evaluation, and conclusion in sequence.
2. AUDIOBOOK CORPUS FOR END-TO-END SPEECH TRANSLATION
The authors augment LibriSpeech with aligned English speech and French translations to create a large audiobook speech-translation corpus. The corpus supports direct AST experiments and ASR and MT baselines.
- Motivation: The corpus addresses the scarcity of large, publicly available speech-to-target-text parallel corpora.Existing examples cited are smaller, low-bandwidth, or not freely available.
- Source corpus: LibriSpeech contributes 1000 hours of speech aligned with English transcriptions as the starting ASR corpus.The augmented English-to-French corpus is described as much larger than existing alternatives.
- Corpus construction: 236 hours of English speech are aligned with French translations at the utterance level.The augmentation automatically aligns French e-books with English LibriSpeech utterances.
- Annotations: Each utterance provides English speech, an English transcription, and two French translations from e-book alignment and transcript machine translation.The English source transcriptions enable both transcript-based and speech-based experiments.
- Evaluation: The study compares direct AST with cascaded ASR-plus-MT and reports ASR and MT results as corpus baselines.The AST task translates English audiobooks into French.
- Splits: The dataset is split into 4 hours of test data, 2 hours of development data, 100 hours of clean training data, and 130 hours of extended training data.Segments were sorted by alignment confidence scores.
3. END-TO-END MODELS
The models use attention-based encoder-decoder architectures for ASR, MT, and AST, sharing encoder or decoder components across tasks. The speech encoder combines convolutional downsampling with bidirectional LSTMs, while the decoder generates French characters using attention.
- Multi-task architecture: The ASR and AST models share an encoder architecture, while the AST and MT models share a decoder architecture.This design supports multi-task training across the three tasks.
- Speech encoder: The speech encoder combines nonlinear feature layers, two convolutional layers, and three bidirectional LSTM layers.The convolutional layers use stride 2 in time and feature dimensions.
- Speech encoder: The convolutional stack reduces the feature sequence to one-quarter of its original time length before recurrent processing.The reduced sequence speeds training because model complexity is quadratic in source length.
- Speech encoder: The final bidirectional LSTM produces annotations by concatenating corresponding forward and backward states.Each annotation has dimensionality 2m.
- Speech encoder: The encoder differs from earlier work by using convolutional downsampling instead of time pooling between LSTM layers.Earlier work used a pyramidal encoder.
- Character-level decoder: The decoder is character-level and uses a conditional LSTM followed by a dense layer.The decoder architecture is shared with the MT model.
- Character-level decoder: Global attention produces a context vector summarizing encoder states, which helps the decoder generate and update its state.The decoder then scores symbols in the target vocabulary.
- Character-level decoder: The generation layer selects the target symbol with the highest score from the target vocabulary.A nonlinear layer and linear projection compute the symbol scores.
4. EXPERIMENTS
Experiments evaluate ASR, MT, and AST on BTEC and Augmented LibriSpeech using encoder-decoder models, reporting test results and training behavior. The results indicate that pre-training and multi-task learning can improve AST, while audiobook translation remains challenging.
- Evaluation: Experiments evaluate ASR and MT on BTEC and Augmented LibriSpeech, using BLEU for translation and WER for recognition.Models are evaluated on development data during training, with the best checkpoint retained for final test evaluation.
- Results: Around 20% BLEU on LibriSpeech shows that translating audiobook novels is particularly challenging.Google Translate is scored as a 22.2% topline.
- Model settings: AST compares cascaded, end-to-end, pre-trained, and multi-task settings, with shared encoder or decoder components enabling multi-task training.The cascaded system combines ASR and MT, while the end-to-end system decodes speech into target text without source transcripts.
- Analysis: Multi-task training improves AST performance but can degrade MT and ASR results because it allocates more updates to AST.Despite this degradation, the speech encoder and text decoder generalize well to other tasks.
- Results: Across BTEC and LibriSpeech, compact end-to-end AST models achieve performance close to cascaded models, although cascaded systems remain best.The LibriSpeech results also support Augmented LibriSpeech as a useful but challenging benchmark for real-speech AST.
- Analysis: Pre-training helps AST converge faster, while the end-to-end model eventually reaches a similarly good solution after three times as many updates.The best development scores for end-to-end, pre-trained, and multi-task models occur at 369k, 129k, and 95k steps, respectively.
5. CONCLUSION
Cascaded ASR–MT models achieve the best results, but end-to-end methods incorporating source-language transcripts come close on the evaluated speech-translation corpora.
- Cascaded ASR and MT models give the best results, while transcript-informed end-to-end methods come close in performance.