Source-linked AI summary
Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation
Alexandre Berard, Olivier Pietquin, Christophe Servan, Laurent Besacier
TL;DR
The paper addresses speech translation systems’ reliance on source-language transcripts. It proposes direct end-to-end speech-to-text translation with an attention-based encoder-decoder, obtaining promising results on a small French-English synthetic corpus, while noting that synthetic data limits the evidence and motivating evaluation on non-synthetic data.
Problem
Existing speech translation systems require source-language transcription, creating a data-collection burden especially for under-resourced or unwritten languages.
Method
The paper trains an attention-based encoder-decoder that translates speech directly into target text without source transcription during learning or decoding.
Results
The model gives promising results on a small French-English synthetic corpus and generalizes well to a new speaker.
Takeaways & Limitations
Direct speech-to-text translation may support alternative data collection by having bilingual speakers utter source-language speech from target-language text.
Takeaways & Limitations
The results are mitigated by synthetic speech with limited variability, and the model remains to be evaluated on non-synthetic data.
Abstract
from arXiv · showhide
This paper proposes a first attempt to build an end-to-end speech-to-text translation system, which does not use source language transcription during learning or decoding. We propose a model for direct speech-to-text translation, which gives promising results on a small French-English synthetic corpus. Relaxing the need for source language transcription would drastically change the data collection methodology in speech translation, especially in under-resourced scenarios. For instance, in the former project DARPA TRANSTAC (speech translation from spoken Arabic dialects), a large effort was devoted to the collection of speech transcripts (and a prerequisite to obtain transcripts was often a detailed transcription guide for languages with little standardized spelling). Now, if end-to-end approaches for speech-to-text translation are successful, one might consider collecting data by asking bilingual speakers to directly utter speech in the source language from target language text utterances. Such an approach has the advantage to be applicable to any unwritten (source) language.
1 Introduction and background
Existing speech translation systems typically require source-language transcription between speech recognition and text translation. This paper targets direct speech-to-text translation without source transcription during learning or decoding.
- Current systems combine source-language ASR with source-to-target MT, making a source transcript mandatory for target text generation.
- Deep neural systems commonly use encoder-decoder architectures that map input symbols or signals into continuous representations before generating output sequences.
- Before this work, direct source-speech-to-target-text translation had received little attention beyond pioneering phone-based attempts and one alignment-focused study.
2 Model
The paper uses attention-based encoder-decoder models for both text and speech translation, adapting the speech model to acoustic feature sequences and long inputs. The decoder generates target words from recurrent states and attention over encoder outputs.
- Description: Both systems are attention-based encoder-decoder neural networks, with separate models for standard text translation and speech translation.
- Encoder-decoder: A bidirectional LSTM encoder produces output sequences, while the decoder is initialized from the encoder and recurrently generates target-side states.
- Decoder: The decoder concatenates an attention vector with its LSTM output, projects the result to target-vocabulary probabilities, and selects the highest-probability word during greedy decoding.
- Attention mechanism: For speech, convolutional attention incorporates previous attention weights so the model can avoid translating the same signal region twice.
- Model settings: The speech model receives 40 MFCCs plus frame energy, uses a three-layer hierarchical bidirectional encoder, and shares a two-layer 256-unit LSTM decoder configuration with the text model.
- Training: Training uses Adam with learning rate 0.001, mini-batches of 64, and dropout rate 0.5 between encoder and decoder LSTM layers.
3 Experimental Settings
Because a sufficiently large speech-translation corpus was unavailable, the experiments use speech synthesized from the French-English BTEC corpus. The data varies across speakers and reserves one speaker for evaluation.
- The authors generate speech data from the small French-English BTEC parallel corpus because no good-sized speech-translation corpus was available.BTEC contains short sentences, a small vocabulary, and multiple references for test evaluation.
- Synthetic speech uses four female and three male voices produced by corpus-based concatenative synthesis rather than parametric synthesis.
- Training combines six speakers, while Agnes is reserved for evaluation to test robustness to inter-speaker variability.
- Speech is segmented into 40 ms frames with 10 ms steps, and each frame is represented by 40 MFCCs plus frame energy.
4 Experiments
The experiments compare end-to-end speech translation with pipeline and perfect-ASR baselines on synthetic French-English BTEC data, including evaluation on an unseen speaker. Results are promising but remain below the pipeline baseline.
- Machine Translation: The machine translation ensemble with five independently trained models and a language model achieved BLEU scores similar to the SMT baseline.The comparison tests whether NMT can work on the small, specialized BTEC corpus.
- Speech Translation: The best speech translation results are behind, but not very far from, a Google Speech API plus specialized SMT baseline.The baseline combines Google Speech ASR with an SMT system trained on BTEC.
- Speech Translation: Agnes, an unseen speaker excluded from training, produced encouraging results without speaker adaptation.The speech models were trained with six speakers; Agnes was not used for training.
- Training: Figure 1 presents attention-model alignments during training for both text and speech models.The models were trained for 20k steps; text training took under 2 hours and speech training roughly 8 hours on one GTX 1070.
5 Conclusion
The paper concludes that its end-to-end architecture gives promising results on a small French-English synthetic corpus and generalizes to a new speaker. The synthetic data limits how broadly these results should be interpreted, motivating evaluation on non-synthetic data.
- Conclusion: The proposed end-to-end speech translation model gives promising results on a small French-English synthetic corpus.The conclusion identifies this as the paper’s central experimental outcome.
- Conclusion: The architecture generalizes quite well to a new speaker despite using no speaker adaptation technique.The conclusion qualifies this result within the reported experiments.
- Limitations: The results are mitigated by the synthetic data’s limited variability, which makes learning easier for the network.The data has little variability except across speakers.
- Future Work: Future work includes evaluating the model on non-synthetic data such as TED talks or human-read audiobooks.The paper notes that TED-talk alignment is not straightforward.