PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Speech-to-Speech Translation Through Unlabeled Text

Xuan-Phi Nguyen, Sravya Popuri, Changhan Wang, Yun Tang, Ilia Kulikov, Hongyu Gong

arXiv:2210.14514v1cs.CLcs.SDeess.AS

TL;DR

Direct S2ST faces a shortage of human-annotated parallel speech, while unlabeled text remains underused as a data source. The paper turns that text into synthetic S2ST training data and applies acoustic effects to it. The method surpasses the prior approach across Spanish-English and English-Spanish test sets, with especially strong gains in low-resource conditions.

  • Problem

    Direct S2ST data is scarce because collecting human-annotated parallel speech is challenging, and existing synthetic-data work had not used the abundant, diverse unlabeled text.

    Source: §1 · p. 1

  • Method

    The method translates cleaned unlabeled text between languages, synthesizes speech from both sides, and applies randomized acoustic effects and noise to the synthetic speech.

    Source: §1 · p. 1

  • Results

    Across multiple test sets, the method surpasses [12] by up to 2 BLEU for Es→En and 1 BLEU for En→Es.

    Source: §4.1 · p. 3

  • Takeaways & Limitations

    Effects-aug is crucial for synthetic data to help, and adding Text-aug provides an extra 1 BLEU improvement over Effects-aug on original S2ST data alone.

    Source: §4.4 · p. 4

Abstract

from arXiv · show

Direct speech-to-speech translation (S2ST) is among the most challenging problems in the translation paradigm due to the significant scarcity of S2ST data. While effort has been made to increase the data size from unlabeled speech by cascading pretrained speech recognition (ASR), machine translation (MT) and text-to-speech (TTS) models; unlabeled text has remained relatively under-utilized to improve S2ST. We propose an effective way to utilize the massive existing unlabeled text from different languages to create a large amount of S2ST data to improve S2ST performance by applying various acoustic effects to the generated synthetic data. Empirically our method outperforms the state of the art in Spanish-English translation by up to 2 BLEU. Significant gains by the proposed method are demonstrated in extremely low-resource settings for both Spanish-English and Russian-English translations.

1. INTRODUCTION

Direct S2ST avoids the latency and error propagation of cascaded translation systems, but scarce parallel speech data limits training. The paper uses unlabeled text to generate prosodically varied synthetic S2ST data and reports improvements over prior results.

1. INTRODUCTION

  • Cascading ASR, MT, or S2T with TTS incurs significant inference latency and risks error propagation, whereas direct S2ST can also translate unwritten languages and dialects.Recent direct systems use speech-to-unit translation with discrete units that encode linguistic and prosodic information.Source: §1 · p. 1
  • Direct S2ST data is scarce because collecting human-annotated parallel speech is challenging, while prior synthetic-data work used available audio but left unlabeled text untapped.Text is more massive and diverse than current speech data, but lacks speaker, speed, pitch, and emotion information.Source: §1 · p. 1
  • The proposed text-based synthetic data and acoustic augmentation outperform the state of the art by up to 2 BLEU and yield significant gains in low-resource setups.The method reaches 35.2 BLEU on CoVoST-2 Es-En and 35.1 BLEU on Europarl-ST En-Es; the paper also reports gains with only 10 hours of S2ST data.Source: §1 · p. 1

2. BACKGROUND

The background describes discrete-unit speech translation as a way to generate shorter sequences for a lightweight vocoder, and situates the method relative to prior synthetic-data augmentation.

2. BACKGROUND

  • HuBERT produces discrete speech units by clustering hidden vectors with a pretrained k-means model, and these units condense semantic and prosodic information.The units are derived from pretrained speech representations.Source: §2 · p. 2
  • Recent S2ST systems generate discrete unit sequences with an S2UT model and use a lightweight vocoder to convert them into audio.The S2UT encoder uses pretrained speech representations, while its decoder is a unit-mBART model; the model is initialized from pretrained components and fine-tuned on speech-to-unit data.Source: §2 · p. 2
  • Popuri et al. [12] augment training with synthetic speech generated from supervised ASR data using pretrained MT and TTS; this work adds unlabeled text as another source.Figure 2 depicts Text-aug data generation and S2U training with Effects-aug.Source: §2 · p. 2, Figure 2 · p. 2

3. METHOD

The method creates synthetic S2ST training data from unlabeled text, then applies randomized acoustic effects and noise to make the generated speech more varied. The process is intended to address the monotonic, single-style prosody of single-speaker TTS data.

3. METHOD

  • The method combines text-derived synthetic data creation with on-the-fly acoustic augmentation of that data during S2U training.The method section frames these as Text-aug data generation and training that augments the synthetic data to be acoustically diverse.Source: §3 · p. 2

3.1. Text-Aug Data Creation

  • Text-aug data is generated by cleaning target-language text, translating it into source-language text with unsupervised CRISS, filtering sentence pairs, and synthesizing speech in both languages.The target speech is converted into reduced units for speech-to-unit training.Source: §3.1 · p. 2

3.2. Effects-Aug Data Augmentation

  • Effects-aug applies randomized acoustic effects and background noise to Text-aug source speech during training to vary its tones and recording conditions.Effects include speed, pitch, and frequency changes, low-pass filtering, and mixing with environmental noise or music; each effect stage applies its perturbation with probability p or leaves the audio unchanged.Source: §3.2 · p. 3

4. EXPERIMENTS

Experiments compare the method with prior systems across Spanish–English, Russian–English, low-resource, and unsupervised S2ST settings, then ablate its augmentation choices. Results include gains over the prior system and especially strong benefits in low-resource settings.

4.1. Spanish-English Speech-to-Speech Translation

  • On average across multiple test sets, the method surpasses [12] by up to 2 BLEU for Es→En and 1 BLEU for En→Es.It also exceeds [12] with extra back-translation data by 1 BLEU on average.Source: §4.1 · p. 3, Table 1 · p. 4

4.2. Low-resource and Unsupervised Translation

  • In the unsupervised setup, the method achieves 17.1 BLEU for Es-En and 18.1 BLEU for En-Es; gains are strongest at 10hr and diminish as natural-speech S2ST data increases to 100hr.The evaluation also uses 50hr of S2ST data.Source: §4.2 · p. 3, Table 2 · p. 4

4.3. Russian-to-English Translation

  • The method achieves up to 34 BLEU gain over standard S2ST on Russian-English, a task described as inherently extremely low-resource.The comparison covers a 10hr subsample and the full 25hr natural S2ST dataset.Source: §4.3 · pp. 3–4, Table 3 · p. 4

4.4. Ablation Studies

  • Effects-aug on Text-aug data is critical, while applying it to S2ST data as well gives only a slight improvement.Source: §4.4 · p. 4
  • Augmenting target-side audio is unnecessary and makes little difference.Source: §4.4 · p. 4
  • Applying both SoX effects and background noise to Text-aug data yields the best results; omitting Effects-aug causes dramatic degradation.Either SoX effects or background noise alone brings a decent gain.Source: Table 4 · p. 4, §4.4 · p. 4
  • Adding Text-aug data offers an extra 1 BLEU improvement over applying Effects-aug only to original S2ST data.Effects-aug alone yields up to 1 BLEU gain over baseline [12].Source: §4.4 · p. 4

5. CONCLUSION

The method combines synthetic S2ST data generated from unlabeled text with online audio augmentation for prosodic diversity. It outperforms the state of the art in Spanish-English S2ST by up to 2 ASR-BLEU and is more beneficial in low-resource and unsupervised setups.

  • Up to 2 ASR-BLEU improvement over the state of the art in Spanish-English S2ST.Source: §5 · p. 4
  • The method generates synthetic S2ST data from unlabeled text and applies online audio augmentation for greater prosodic diversity.Source: §5 · p. 4
  • The method is more beneficial in low-resource and unsupervised setups.Source: §5 · p. 4
Loading 2210.14514v1…