Source-linked AI summary
Tied Multitask Learning for Neural Speech Translation
Antonios Anastasopoulos, David Chiang
TL;DR
Low-resource speech transcription and translation need models that exploit limited data and useful intermediate representations. The paper introduces tied multitask decoders with transitivity and invertibility regularization, improving low-resource speech tasks and word discovery, while text translation shows no similar improvement.
Problem
Extremely small speech datasets make it difficult to jointly support transcription and translation, motivating multitask models that can use both task signals.
Method
The paper ties multitask sequence-to-sequence decoders so the second decoder receives first-decoder representations, and regularizes attention for transitivity and invertibility.
Results
The model improves low-resource speech transcription and translation against baselines and previous multitask architectures, with gains also reported for word discovery.
Takeaways & Limitations
Tied multitask learning with these regularizers improves performance across the paper’s low-resource speech tasks and word discovery setting.
Takeaways & Limitations
The extremely low-resource experiments still produce speech translations with substantial room for improvement, and larger datasets may require a more sophisticated speech encoder.
Abstract
from arXiv · showhide
We explore multitask models for neural translation of speech, augmenting them in order to reflect two intuitive notions. First, we introduce a model where the second task decoder receives information from the decoder of the first task, since higher-level intermediate representations should provide useful information. Second, we apply regularization that encourages transitivity and invertibility. We show that the application of these notions on jointly trained models improves performance on the tasks of low-resource speech transcription and translation. It also leads to better performance when using attention information for word discovery over unsegmented input.
1 Introduction
The paper targets speech transcription and translation in extremely small-data settings, improving multitask learning by passing higher-level representations between decoders and regularizing attention for transitivity and invertibility.
- Endangered-language documentation collects spoken resources with translations in a high-resource language, making the resource interpretable.
- Extremely small datasets motivate jointly training speech transcription and translation so both tasks can contribute learning signals.
- The proposed tied multitask architecture gives the second decoder information from both the encoder and the first decoder.
- Attention regularization encourages transitivity and invertibility, producing further performance improvements.
- The experiments cover low-resource speech transcription and translation, word discovery on unsegmented input, and high-resource text translation.Low-resource speech experiments include Spanish-English, Mboshi-French, and Ainu-English.
- Up to 5% character error rate improvement occurs in transcription, up to 2.8% character-level BLEU improvement in translation, and about 3% F-score improvement for word discovery tokens and types.The text translation experiments did not show similar improvements.
2 Model
The model uses an attentional encoder-decoder framework with separate task decoders, then extends standard multitask and cascade designs into a triangle architecture that combines encoder and first-decoder information.
- The sequence-to-sequence model consists of a recurrent encoder, attention mechanism, and recurrent decoder.
- The encoder maps input words or feature frames to input states, attention maps those states to context vectors, and the decoder produces output states and word probabilities.
- Standard multitask learning uses a shared encoder with separate attentions and decoders for two output sequences.
- In a cascade, the second decoder attends only to the first decoder’s output states.
- The triangle model gives the second decoder two attentions, over encoder input states and first-decoder output states, whose context vectors are concatenated.
3 Learning and Inference
Training maximizes a weighted joint score over sentence triples, optionally regularizes attention for transitivity and invertibility, and uses two-phase beam search to decode transcription and translation candidates.
- The model is trained on sentence triples and represents each triple’s score as a loglinear interpolation of the two decoder probabilities.The task-weight parameter λ is set to 0.5 in all experiments.
- The training objective sums sentence-triple scores over the training data.
- An optional regularization term encourages attention mechanisms to satisfy transitivity and invertibility principles.
- Regularization: The transitivity regularizer encourages source-to-intermediate-to-target alignments to preserve a direct source-to-target relationship.Its weight is λtrans = 0.2.
- Regularization: The invertibility regularizer encourages translator and reconstructor attentions to act as inverses of each other.Its weight is λinv = 0.2.
- Decoding: Two-phase beam search first generates transcription candidates and hidden states, then generates translations for each candidate and selects the highest-scoring pair.Inference uses beam size 4 for each decoder.
4 Speech Transcription and Translation
The models are evaluated on low-resource speech transcription and translation across Spanish-English, Ainu-English, and Mboshi-French, using small and linguistically diverse datasets. The triangle model improves over baseline and standard multitask systems on both tasks in almost all cases, while performance varies with data size and speaker diversity.
- Data and settings: The evaluation covers Spanish-English, Ainu-English, and Mboshi-French speech transcription and translation, including telephone conversations, Ainu narratives, and Mboshi parallel utterances.The Spanish corpus contains about 20 hours of audio and more than 240 speakers; Ainu contains about 2.5 hours, and Mboshi about 4.4 hours.
- Models and baselines: Table 2 compares the proposed models with pivot, direct speech-to-text, standard multitask, and gold-transcription skyline systems.The pivot system translates ASR character output, while the standard multitask system jointly models two outputs with a shared encoder and separate attentions and decoders.
- Results: The triangle model outperforms all baselines, including the standard multitask model, on all three datasets.On Ainu-English, its translations are comparable to the skyline model evaluated with gold Ainu transcriptions.
- Data effects: Character error rates increase with the number of speakers, while character-level BLEU scores increase with the amount of training data.The paper attributes the BLEU trend to the translation decoder learning a more informed character-level language model with more data.
- Limitations: The models underperform a larger-data reference on CALLHOME by almost 9 word-level BLEU points, and the authors report that the datasets are insufficient for high-quality speech translation.The reference model has significantly more parameters and is trained on 10 times more data.
5 Word Discovery
The paper uses attention matrices from jointly trained reconstruction models to discover Mboshi word boundaries from unsegmented phonetic input. The invertibility regularizer produces more informed attentions and improves type-level word discovery.
- 5.3 Results: The triangle model’s A2 attention contains 34% correct attention mass, compared with 21% for the multitask model.The figure attributes the higher correct attention mass to the transitivity regularizer.
- 5 Word Discovery: Word discovery projects French word boundaries onto unsegmented Mboshi phonetic transcriptions using attention weights as soft alignments.The models are trained on unsegmented Mboshi phonetic sequences paired with word-level French translations.
- 5 Word Discovery: The evaluation computes precision, recall, and F-score at both token and type levels over the discovered segmentation.The best results are reported in Table 3.
- 5.3 Results: The study could not reproduce the significant gains previously reported for the reverse model, although its base and reverse singletask models outperformed the reimplemented baseline.This comparison concerns the authors’ reimplementation of prior models.
- 5.3 Results: Combining attention matrices from a jointly trained reconstruction model with invertibility regularization improves type-level word discovery.The regularizer is key: reconstruction models without it did not show improvements.
6 Negative Results: High-Resource Text Translation
In high-resource Europarl text translation, the proposed reconstruction and triangle models generally do not outperform simpler baselines. Regularizers often improve their variants, but the strongest results remain with baseline single-task or simple multitask models.
- 6.1 Data: The Europarl evaluation uses 1,450,890 English–French–German three-way parallel sentences across all six translation directions.The data use BPE representations and separate development and test sets.
- 6.3 Results: Across all six directions, baseline single-task or simple multitask models achieve the best accuracy in every case.Some reconstruction or triangle models are not statistically significantly different from the best model for English–German.
- 6.3 Results: The authors attribute the lack of advantage to lower-level encoder representations already containing sufficient information for linguistically close language pairs, without inference search errors.This is presented as the authors’ explanation for the high-resource results.
- 6.3 Results: Unlike Tu et al. (2017), the experiments do not observe significant reconstruction-model improvements.The authors list differences in subword versus word units, optimization, parameter count, and language-pair diversity as possible explanations.
- 6.3 Results: The proposed regularizers often increase performance: invertibility helps in 3 of 6 cases, while transitivity helps in 9 of 12 source-target combinations.Some transitivity improvements are statistically significant, including French-to-English with English as the intermediate target.
7 Related Work
Related work spans speech translation, attention-based word discovery, multitask learning, and reconstruction architectures. This paper is especially close to prior multitask speech translation and reconstruction-decoder models, while targeting much smaller endangered-language datasets.
- Speech Translation: Earlier speech-translation work used ASR outputs or lattices, while later studies explored direct speech translation and attention-based word discovery.Examples include lattice-based translation, synthesized-speech translation, end-to-end audiobook translation, and attention-based segmentation.
- Multitask Learning: Multitask learning has been applied to parsing and translation, CTC-and-attention speech transcription, multilingual translation, and neural speech recognition.These studies establish multitask learning as a broader precedent across machine learning and NLP.
- Multitask Learning: Weiss et al. jointly trained speech transcription and English translation with decoders sharing an encoder, using roughly 163 hours of Spanish speech data.Their model was substantially larger, with eight encoder layers and four layers per decoder.
- Reconstruction Models: Tu et al. added a reconstruction decoder attending to the translation decoder’s final states and reported gains for Chinese–English translation.Their approach was mainly inspired by auto-encoders.
8 Conclusion
The paper concludes that tying the second task to higher-level representations from the first decoder improves low-resource speech transcription, translation, and attention-based word discovery. It also identifies endangered-language documentation as a potential application, while noting that sentence triples may not always be available.
- 8 Conclusion: The proposed architecture supplies the second task with higher-level representations produced by the first task decoder.This distinguishes it from traditional multitask architectures that share encoder-level information.
- 8 Conclusion: The model improves low-resource speech transcription and translation over singletask models and traditional multitask architectures.Adding transitivity and invertibility regularizers yields further improvements on all low-resource tasks.
- 8 Conclusion: The methods could support endangered-language documentation where transcribing large audio collections is impractical.The authors propose extending the approach to settings where some audio is transcribed and other audio is translated, without sentence triples.