PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation

Elizabeth Salesky, Matthias Sperber, Alan W Black

arXiv:1906.01199v1cs.CLcs.SDeess.AS

TL;DR

End-to-end speech translation has commonly used frame-level features, which yield longer, sparser sequences, and alternative speech representations had not been compared. This paper averages frames with the same phoneme label to make shorter phoneme-like input sequences. The approach improves results across two language pairs and multiple data sizes, with up to 5.2 BLEU improvement on the full Fisher dataset.

  • Problem

    End-to-end speech translation had not compared different speech input representations, although frame-level features yield longer, sparser sequences than text equivalents.

    Source: §1 · p. 1

  • Method

    The method generates phoneme labels and averages consecutive same-label frames into phoneme-like feature vectors, using a cross-lingual approach for low-resource Mboshi.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    Up to 5.2 BLEU improvement was reported on the full Fisher dataset, with relative performance improvements of 13% across all data sizes.

    Source: §5.2 · p. 3

  • Takeaways & Limitations

    The authors conclude that this naive higher-level representation can be more effective and efficient than frame-level features across two unrelated language pairs and varied resource conditions, without adding parameters.

    Source: §6 · p. 5

    The comparison of monolingual HMM and cross-lingual CTC alignments needs a more robust study, including more test languages.

    Source: §5.3 · p. 5

Abstract

from arXiv · show

Previous work on end-to-end translation from speech has primarily used frame-level features as speech representations, which creates longer, sparser sequences than text. We show that a naive method to create compressed phoneme-like speech representations is far more effective and efficient for translation than traditional frame-level speech features. Specifically, we generate phoneme labels for speech frames and average consecutive frames with the same label to create shorter, higher-level source sequences for translation. We see improvements of up to 5 BLEU on both our high and low resource language pairs, with a reduction in training time of 60%. Our improvements hold across multiple data sizes and two language pairs.

1 Introduction

Speech translation typically represents input with fixed frame-level features, unlike text translation’s discrete subword units, but different speech representations had not been compared. This work explores compressed phoneme-informed representations and reports improved performance and training efficiency across resource conditions.

1 Introduction

  • Text translation commonly segments words into subword units, whose more frequent units can improve generalization to rare word forms (Sennrich et al., 2016).Input representation has also been linked to performance and the amount of training data required (Sennrich et al., 2016; Salesky et al., 2018; Cherry et al., 2018).Source: §1 · p. 1
  • Frame-level speech features form longer, sparser sequences than text equivalents, and prior end-to-end speech translation work had not compared alternative speech representations.Earlier work reduced sequence lengths through fixed-length downsampling, while phoneme representations had also been explored as discrete units.Source: §1 · p. 1
  • The paper explores higher-level continuous speech embeddings, motivated by variable-length phonemes and prior work on discrete phonemic representations and term discovery.Prior work includes Wilkinson et al. (2016), Bansal et al. (2017), Adams et al. (2016), Kamper et al. (2016), Dalmia et al. (2018b), and Chung and Glass (2018).Source: §1 · p. 1
  • The method generates phoneme labels and averages consecutive same-label frames into phoneme-like features, improving performance and reducing training time across the evaluated conditions.The study uses Fisher Spanish-English and low-resource Mboshi-French data, with a high-resource model applied cross-lingually to label Mboshi speech.Source: §1 · p. 1

2 Method

The method turns consecutive speech frames with the same phoneme label into one feature vector, producing variable-length phoneme-like sequences. It uses recognizer-based alignments, with a cross-lingual CTC option for the low-resource setting.

2 Method

  • Frame-level MFCC and filterbank features create long, repetitive sequences that recurrent models take many examples to learn to model.The paper motivates higher-level representations as a way to create shorter input sequences and improve training efficiency.Source: §2 · p. 1
  • Figure 1’s example shows 50 frame-level features versus 8 phoneme alignments, and the reported average sequence-length reduction is ∼80%.The figure illustrates the shorter source sequences produced by the method.Source: Figure 1 · p. 2
  • Consecutive frames with the same phoneme-like label are averaged into one feature vector, with the number of frames varying by unit.The approach uses a trained speech recognizer and alignment.Source: §2 · p. 2
  • The HMM/DNN triphone aligner uses 50 labels, including silence, noise, and laughter variants, to group consecutive frames within each utterance.The input features are 40-dimensional Mel filterbanks with per-speaker mean and variance normalization.Source: §2 · p. 2
  • For Mboshi, a CTC model trained on 6000 hours of English generates labels without language-specific training data, and consecutive same-label frames are averaged.The label set contains 40 phonemes and a blank label; three spliced frame-level features yield output labels spanning three frames.Source: §2 · p. 2

3 Model Architecture

The paper uses an attention-based encoder-decoder architecture adapted to fit available training resources. Its encoder uses network-in-network projections for temporal downsampling instead of additional convolutional layers.

3 Model Architecture

  • The encoder-decoder models use an attention-based sequence-to-sequence architecture and can be trained in less than 5 days on one GPU.The models have 512 hidden units throughout and are implemented in xnmt.Source: §3 · p. 2
  • A 3-layer BiLSTM encoder uses network-in-network projections for 4× temporal downsampling instead of additional convolutional layers.The projections add depth with fewer parameters, following prior sequence-to-sequence ASR work (Zhang et al., 2017; Sperber et al., 2018).Source: §3 · p. 2
  • The model uses one-hidden-layer MLP attention, 64-dimensional target embeddings, and a single decoder hidden layer.Target text is lowercased and punctuation is removed except apostrophes.Source: §3 · p. 2

4 Datasets

The experiments use Fisher Spanish-English as the higher-resource setting and Mboshi-French as the low-resource setting. The datasets differ in corpus size and in how development and test data are defined.

4 Datasets

  • Fisher Spanish-English uses 160 hours of telephone speech across 138K utterances, with crowdsourced translations and standard development and test sets.Each standard evaluation set has approximately 4K utterances, and four reference translations score development and test.Source: §4 · p. 2
  • The Fisher Spanish corpus covers multiple Spanish dialects and is translated via crowdsourcing.The study does not use dev2.Source: §4 · p. 2
  • Mboshi-French provides less than 5 hours of speech, with 4616 training utterances and a 500-utterance development set.Mboshi is a Bantu language spoken in the Republic of Congo, with approximately 160K speakers (Godard et al., 2017).Source: §4 · p. 2
  • Because Mboshi-French has no designated test set, 200 training utterances were randomly held out for development, and the designated development set was used for testing.This follows the evaluation setup in Bansal et al. (2018b).Source: §4 · p. 2

5 Results

Phoneme-like input improves Spanish–English speech translation across data sizes while shortening training, and analysis finds benefits in ASR and low-resource Mboshi–French. The results also identify open questions about alignment comparisons.

5.1 Baseline

5.2 Frames vs Phonemes

  • 13% relative BLEU improvement across data sizes, up to 5.2 BLEU on the full dataset, and 61% average training-time reduction with phoneme-like input.On the full dataset, training took 39.5 hours rather than 118.2.Source: Table 2 · p. 3, §5.2 · p. 3
  • Fixed-stride downsampling lowers 40-hour BLEU, while variable-length phoneme-informed reduction improves it by about 2 BLEU.Stride 2 changes dev/test BLEU from 19.5/17.4 to 17.0/15.6; stride 3 yields 13.7/11.8.Source: §5.2 · p. 3

5.3 Analysis

  • Phoneme averaging reduces source lengths by 79%, producing final encoder states averaging 22, closest to 1k BPE targets at 14 rather than character targets at 50.The 1k BPE model performs best; the authors suggest more similar source and target lengths may boost performance.Source: §5.3 · pp. 3–4, Table 3 · p. 3
  • ASR shows similar gains: about 18% relative improvement across all three dataset sizes and up to 9 absolute WER on 40- and 20-hour sets.The authors interpret the ASR results as suggesting the representation is generally more effective, not only useful for speech translation.Source: Table 4 · p. 4, §5.3 · p. 4
  • Mboshi–French results improve with phoneme-like input labeled by an English model, but small-set scores are less stable.The authors report similar levels of improvement to prior work that used additional pretraining data, despite using an unadapted foreign-language model.Source: Table 5 · p. 4, §5.3 · p. 4
  • Phoneme-boundary inputs improve dev BLEU in early epochs across data conditions, indicating more efficient learning.Figure 2 plots single-reference dev BLEU over training on 1,000 lines.Source: §5.3 · p. 4, Figure 2 · p. 4
  • The alignment comparison is limited: HMM alignments are monolingual and CTC alignments cross-lingual, and the authors call for more robust comparisons across languages.The CTC model leaves some frames blank, reducing the frames averaged, though the center of mass often remains the same.Source: §5.3 · p. 5

6 Conclusion

The paper shows that higher-level speech representations can outperform and be more efficient than frame-level features for speech translation. It compares two representations across unrelated language pairs and resource conditions, using supervised and cross-lingual methods.

6 Conclusion

  • The phoneme-like representations are more effective and efficient than traditional frame-level features for speech translation.The comparison covers two unrelated language pairs and differently resourced conditions.Source: §6 · p. 5
  • The approach adds no parameters, and the experiments use both supervised alignment and a cross-lingual method in the low-resource case.Source: §6 · p. 5
  • The results motivate future work on learned speech representations, with performance in lower-resource settings remaining a goal if additional parameters are introduced.Source: §6 · p. 5

A.1 Encoder Downsampling Procedure

The encoder uses LSTM/NiN blocks to downsample in time, reaching the same total factor as the cited speech-translation systems while adding depth with fewer parameters.

A.1 Encoder Downsampling Procedure

  • Unlike the cited systems’ strided convolutional layers, the LSTM/NiN encoder uses network-in-network projections for temporal downsampling.Source: §A.1 · p. 6
  • Each LSTM/NiN block projects adjacent timestep inputs together, downsampling by 2; a final LSTM brings the encoder to three BiLSTM layers and 4× total downsampling.Source: §A.1 · p. 6
  • The LSTM/NiN blocks provide added depth with fewer parameters than the described alternatives.Source: §A.1 · p. 6

A.2 Training Procedure

Training follows the cited procedure, with specified dropout, decoding, batching, and optimization settings.

A.2 Training Procedure

  • Training uses variational recurrent dropout of 0.2 and target character dropout of 0.1, with label smoothing and target embedding norm fixed to 1.Source: §A.2 · p. 6
  • Inference uses beam size 15 and length-normalization exponent 1.5, while dynamic batching gives an average batch size of 36.Source: §A.2 · pp. 6–7
  • Adam starts at learning rate 0.0003; validation BLEU triggers 0.5 decay after 10 epochs initially and 5 after the first decay.The models omit L2 weight decay and Gaussian noise, lowercase and remove punctuation except apostrophes, and exclude utterances longer than 1500 frames.Source: §A.2 · p. 7
Loading 1906.01199v1…