PaperlayerView exact revision on arXiv

Source-linked AI summary

TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation

Rongjie Huang, Jinglin Liu, Huadai Liu, Yi Ren, Lichao Zhang, Jinzheng He, Zhou Zhao

arXiv:2205.12523v2cs.CLcs.SDeess.AS

TL;DR

Textless S2ST faces indeterministic speech units caused by acoustic variation, while autoregressive decoding limits parallelism. TranSpeech uses bilateral perturbation and non-autoregressive mask-predict decoding. It reports higher BLEU than baseline textless S2ST models and up to 21.4x speedup over an autoregressive baseline.

  • Problem

    Acoustic variation makes discrete units indeterministic for same-content speech, and autoregressive systems predict each unit from previously generated units rather than using parallel decoding.

    Source: §1 · p. 2

  • Method

    TranSpeech uses bilateral perturbation—style normalization and information enhancement—to learn linguistic representations, then applies non-autoregressive mask-predict decoding.

    Source: §1 · p. 2

  • Results

    2.9 BLEU average improvement over baseline textless S2ST models and up to 21.4x speedup over the autoregressive baseline are reported.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The reported results show that TranSpeech combines improved translation quality with parallel decoding that reduces inference latency.

    Source: §1 · pp. 2–3

    The model assumes that target sequence length can be modeled with a separate conditional distribution.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

Direct speech-to-speech translation (S2ST) with discrete units leverages recent progress in speech representation learning. Specifically, a sequence of discrete representations derived in a self-supervised manner are predicted from the model and passed to a vocoder for speech reconstruction, while still facing the following challenges: 1) Acoustic multimodality: the discrete units derived from speech with same content could be indeterministic due to the acoustic property (e.g., rhythm, pitch, and energy), which causes deterioration of translation accuracy; 2) high latency: current S2ST systems utilize autoregressive models which predict each unit conditioned on the sequence previously generated, failing to take full advantage of parallelism. In this work, we propose TranSpeech, a speech-to-speech translation model with bilateral perturbation. To alleviate the acoustic multimodal problem, we propose bilateral perturbation (BiP), which consists of the style normalization and information enhancement stages, to learn only the linguistic information from speech samples and generate more deterministic representations. With reduced multimodality, we step forward and become the first to establish a non-autoregressive S2ST technique, which repeatedly masks and predicts unit choices and produces high-accuracy results in just a few cycles. Experimental results on three language pairs demonstrate that BiP yields an improvement of 2.9 BLEU on average compared with a baseline textless S2ST model. Moreover, our parallel decoding shows a significant reduction of inference latency, enabling speedup up to 21.4x than autoregressive technique. Audio samples are available at \url{https://TranSpeech.github.io/}

1 INTRODUCTION

TranSpeech addresses acoustic multimodality in textless S2ST with bilateral perturbation and introduces non-autoregressive decoding. It reports improved translation quality and substantially faster inference than the respective baselines.

1 INTRODUCTION

  • Acoustic variation makes same-content speech produce different discrete units, creating indeterministic training targets that hinder translation accuracy and complicate parallel decoding.The cited variation includes speaker identity, rhythm, pitch, and energy.Source: §1 · p. 2
  • Bilateral perturbation combines style normalization and information enhancement to train representations focused on linguistic content rather than acoustic variation.The method fine-tunes a self-supervised speech representation model with CTC loss, using normalized targets and acoustically perturbed speech.Source: §1 · p. 2
  • 2.9 BLEU average improvement over baseline textless S2ST models and up to 21.4x speedup over the autoregressive baseline are reported.The parallel decoder can outperform competing systems in as few as two iterations; advanced decoding choices trade translation quality for speed by up to 3 BLEU points.Source: §1 · pp. 2–3
  • TranSpeech introduces non-autoregressive S2ST that repeatedly masks and predicts unit choices, while knowledge distillation creates a less noisy, more deterministic corpus.The proposed mask-predict algorithm is intended to speed inference.Source: §1 · pp. 2–3

2 BACKGROUND: DIRECT SPEECH-TO-SPEECH TRANSLATION

Prior direct S2ST systems vary in their use of phoneme and transcript resources. Textless systems use self-supervised representations, but those representations can retain acoustic variation beyond speaker identity.

2 BACKGROUND: DIRECT SPEECH-TO-SPEECH TRANSLATION

  • Translatotron 2 uses an auxiliary target phoneme decoder to promote translation quality but requires phoneme data, while UWSpeech discards target-language transcripts but requires paired speech and phoneme corpora.Translatotron is described as the first direct S2ST model, with reasonable translation accuracy and speech naturalness.Source: §2 · p. 3
  • Self-supervised textless S2ST can avoid text data, but learned representations may include acoustic conditions alongside linguistic content.A speaker-invariant approach constrains speaker identity, while content, rhythm, pitch, and energy remain lumped together.Source: §2 · p. 3
  • TranSpeech tunes a pretrained acoustic model with bilateral perturbation to produce units agnostic to acoustic conditions and develops a challenging non-autoregressive S2ST technique for low-latency applications.The passage situates this approach alongside acoustic-model reprogramming work, including multilingual spoken command recognition.Source: §2 · p. 3

3 SPEECH ANALYSIS AND BILATERAL PERTURBATION

The paper frames acoustic multimodality as variation in speech units despite unchanged linguistic content, then uses bilateral perturbation to produce more deterministic representations. Its two stages normalize acoustic style and enhance acoustic variation while preserving linguistic content.

3.1 ACOUSTIC MULTIMODALITY

  • Same-content speech can yield different self-supervised representations because the representations include acoustic information, making speech-to-unit targets indeterministic.The paper identifies speaker identity, rhythm, pitch, and energy as acoustic conditions relevant to this issue.Source: §3.1 · p. 3

3.2 SPEECH ANALYSIS

  • Speech variation is decomposed into linguistic content and acoustic conditions, including speaker identity, rhythm, pitch, and energy.Figure 2(a) identifies random resampling and a pitch-shifting chain function; Figure 2(b) labels positional encoding and masked-position cross-entropy training.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 BILATERAL PERTURBATION

  • Bilateral perturbation uses CTC fine-tuning with perturbed speech and normalized targets to disentangle acoustic variation and generate deterministic representations from linguistic content.The method focuses on rhythm, pitch, and energy variation in a single-speaker scenario, given prior work on speaker-invariant representations.Source: §3.3 · p. 4

3.3.1 OVERVIEW

  • The overview pairs acoustically perturbed speech with acoustic-agnostic pseudo text as input and target for CTC training, encouraging the model to learn linguistic content across acoustic variation.The resulting deterministic representations are intended to reduce multimodality and promote speech-to-unit translation.Source: §3.3.1 · p. 4

3.3.2 STYLE NORMALIZATION

  • Style normalization averages pitch and energy across the original dataset, removes style-specific information, and uses the self-supervised model to encode normalized speech as CTC targets.These targets are described as acoustic-agnostic pseudo text.Source: §3.3.2 · pp. 4–5

3.3.3 INFORMATION ENHANCEMENT

  • Information enhancement perturbs rhythm, pitch, and energy while preserving linguistic content, using random resampling and pitch, formant, frequency-shaping, and waveform-domain perturbations.The resulting acoustic variation is intended to encourage deterministic representations focused on linguistic content.Source: §3.3.3 · p. 5

4 TRANSPEECH

TranSpeech converts target speech into discrete units with BiP-tuned HuBERT, translates source speech into units, and uses a separately trained unit vocoder to synthesize waveform. The section describes its Conformer/NAR architecture, knowledge distillation, iterative masking, and decoding options.

4.1 ARCHITECTURE

  • BiP eases acoustic multimodality, enabling NAR S2ST with conditionally independent unit predictions and a length predictor; the encoder uses Conformer blocks.The length predictor pools encoder outputs; attention uses relative sinusoidal positional encoding.Source: §4.1 · pp. 5–6

4.2 LINGUISTIC MULTIMODALITY

  • Knowledge distillation uses an autoregressive teacher to build a less noisy, more deterministic sampled translation corpus for NAR training.The corpus addresses linguistic multimodality from multiple valid translations for the same source utterance.Source: §4.2 · p. 6

4.3 MASK-PREDICT

  • Mask-predict repeatedly refines unit choices; decoding predicts target length, masks all units initially, then remasks low-confidence units with a linearly decaying count.Training samples the number of masked units uniformly from 1 to N and applies label-smoothed cross-entropy to masked units, plus cross-entropy for target-length prediction.Source: §4.3 · p. 6

4.4 ADVANCED DECODING CHOICES

  • Target-length beam decoding evaluates top-K length candidates in parallel and selects the sequence with the highest average log probability; noisy parallel decoding uses an autoregressive teacher to identify the best translation.Batched computation avoids distinctly increasing decoding time for length-beam candidates.Source: §4.4 · p. 6

5 EXPERIMENTS

Experiments compare TranSpeech with direct and cascaded systems on three language pairs, then examine decoding speed, translation examples, and component ablations. Results report gains from bilateral perturbation and faster parallel decoding.

5.1 EXPERIMENTAL SETUP

  • The benchmark uses CVSS-C on Fr-En, En-Es, and En-Fr, evaluates translation accuracy with BLEU, and measures latency on one V100 GPU.The setup compares direct ASR, direct TTS, an S2T+TTS cascade, and textless S2ST baselines.Source: §5.1 · p. 7

5.2 TRANSLATION ACCURACY AND SPEECH NATURALNESS

  • 2.9 BLEU points: bilateral perturbation improves S2ST over the baseline, while TranSpeech with two mask-predict iterations achieves a new textless S2ST state of the art.The comparison is among systems in Table 1; the paper reports that only two iterations are necessary for the new state of the art.Source: §5.2 · p. 7, Table 1 · p. 8

5.3 DECODING SPEED

  • Up to 21.4x speedup: TranSpeech’s non-autoregressive latency is nearly constant for typical input lengths, unlike the linear latency of autoregressive baselines.The paper also reports BLEU 18.39 with a 253% speedup for a quality-retaining operating point.Source: §5.3 · pp. 7–8, Figure 3 · p. 8

5.4 CASE STUDY

  • In Fr-En examples, original-unit models produce noisy, incomplete translations; bilateral perturbation largely alleviates this, and advanced decoding improves accuracy.TranSpeech with more iterations and noisy parallel decoding produces translations of similar quality to the autoregressive basic conformer that are more literal.Source: §5.4 · pp. 8–9, Table 2 · p. 9

5.5 ABLATION STUDY

  • Style normalization and information enhancement both improve performance jointly, while replacing relative positional encoding with vanilla encoding degrades translation accuracy.The ablations examine bilateral perturbation and conformer architecture designs.Source: §5.5 · p. 9, Table 3 · p. 9

6 CONCLUSION

TranSpeech combines bilateral perturbation with non-autoregressive mask-predict decoding for direct speech-to-speech translation. The conclusion reports improved BLEU and substantially faster inference than baseline textless and autoregressive systems.

  • BiP improves BLEU by 2.9 on average over baseline textless S2ST, while TranSpeech reaches up to 21.4x faster sampling than the autoregressive baseline.The paper reports that as few as two iterations generate samples that outperform competing systems.Source: §6 · p. 9
  • Bilateral perturbation uses style normalization and information enhancement to learn linguistic information from acoustically variant speech and generate more deterministic representations.The conclusion says reduced acoustic multimodality makes non-autoregressive generation possible.Source: §6 · p. 9
  • TranSpeech applies mask-predict decoding for non-autoregressive generation and knowledge distillation to construct a less noisy translation corpus.Source: §6 · p. 9

Appendices

The appendices review self-supervised speech representations, perturbation-based speech reprogramming, and non-autoregressive sequence generation. They position TranSpeech’s perturbation approach and direct S2ST focus within those areas.

A.1 SELF-SUPERVISED REPRESENTATION LEARNING

  • Self-supervised speech models often reconstruct or predict unseen signals, retaining factors unrelated to linguistic content, including acoustic conditions.The appendix names Wav2Vec 2.0 and HuBERT as examples of self-supervised representation learning approaches.Source: §A.1 · p. 14

A.2 PERTURBATION-BASED SPEECH REPROGRAMMING

  • Prior perturbation approaches alter acoustic information flow; TranSpeech proposes bilateral perturbation with style normalization and information enhancement to perturb speech conditions.The cited examples include SpeechSplit, AutoPST, and NANSY.Source: §A.2 · p. 14

A.3 NON-AUTOREGRESSIVE SEQUENCE GENERATION

  • Autoregressive decoding limits inference parallelism, while masked-language-modeling approaches refine translations non-autoregressively; direct S2ST remains relatively overlooked in NAR generation.Source: §A.3 · p. 14

B MODEL ARCHITECTURES

The appendix lists TranSpeech’s model hyperparameters in Table 4.

B MODEL ARCHITECTURES

C IMPACT OF INDETERMINISTIC TRAINING TARGET

Acoustic variation changes the units produced by a pretrained SSL model, while bilateral perturbation reduces this variation in the units. UER measures this acoustic sensitivity across original and perturbed speech.

C IMPACT OF INDETERMINISTIC TRAINING TARGET

  • With pretrained SSL, acoustic dynamics produce unit error rates up to 22.7% for rhythm perturbations.The units derived from speech with the same content can therefore differ across acoustic conditions.Source: C IMPACT OF INDETERMINISTIC TRAINING… · p. 15
  • BiP reduces UER by up to 82.8% for energy perturbations, indicating more deterministic units referring to linguistic content.Source: C IMPACT OF INDETERMINISTIC TRAINING… · p. 15
  • Table 5 averages UER between units from original and perturbed speech to measure acoustic-agnostic representation of linguistic content.Source: Table 5 · p. 15

D EVALUATION ON SPEECH QUALITY

TranSpeech’s speech quality evaluation reports SNR and MOS scores competitive with baseline systems. Its information enhancement stage perturbs acoustic features while preserving linguistic content.

D EVALUATION ON SPEECH QUALITY

E INFORMATION ENHANCEMENT

F VISUALIZATION OF ACOUSTIC-PERTURBED SPEECH SAMPLES

The figures visualize perturbed speech through spectrograms and pitch contours while indicating that linguistic content remains unchanged. Figure 5 marks altered units in red.

F VISUALIZATION OF ACOUSTIC-PERTURBED SPEECH SAMPLES

  • Figure 4 shows spectrogram and pitch contours for a single-perturbed condition, with random resampling and a pitch-shifting chain function identified.Source: Figure 4 · p. 17
  • Figure 5 shows a perturbed acoustic condition for the utterance “really interesting work,” with linguistic content unchanged.Source: Figure 5 · p. 17
  • Figure 5 prints altered units in red above the spectrogram.Source: Figure 5 · p. 17
Loading 2205.12523v2…