Learning Shared Semantic Space for Speech-to-Text Translation

1 Introduction

• End-to-end ST still struggles to match cascaded systems because speech and text differ in coding attributes and sequence length, while ST has less parallel data than MT.The cited differences include pitch, volume, and intonation versus words, affixes, and punctuation, and thousands of time frames versus tens of words.Source: §1 · p. 1
• Chimera projects speech and text features into a shared semantic space, unifying ST and MT workflows so MT corpora can augment ST training.Speech-text pairs can also align the semantic memories of the two modalities.Source: §1 · p. 1
• The authors motivate shared text-speech representations with evidence that speech and visual-text processing streams converge in brain regions associated with language activities.They cite neuroimaging work on convergence sites and their relation to reading skills, lexical and semantic recognition, and speech generation.Source: §1 · pp. 1–2
• Only a few prior studies explored unified text-speech representations, which the authors say may reflect the previously described difficulties and marginal improvements.The cited exceptions are Indurthi et al. (2020) and Liu et al. (2020).Source: §1 · p. 2
• 27.1 BLEU on MuST-C EN-DE surpasses the best reported result by up to +1.9 BLEU, with state-of-the-art results across eight directions on MuST-C and Augmented LibriSpeech.The authors also report ablations, design analyses, and visualizations of the learned semantic space.Source: §1 · p. 2
• A bi-modal contrastive training task aligns the two modalities’ semantic memories, alongside Chimera’s state-of-the-art MuST-C performance and modality-agnostic representations.The paper identifies these as its principal contributions.Source: §1 · p. 2

End-to-end ST

• End-to-end ST work applies pretraining, multi-task, meta-learning, and curriculum learning, and augments training with synthesized data from ASR and MT corpora.Knowledge distillation and model adaptation are also reported as effective approaches.Source: §2 · p. 2
• Several prior end-to-end ST approaches train on ASR or MT data but lack modules that semantically bridge audio and text, risking modality mismatch.The cited works most related to Chimera are Indurthi et al. (2020), Le et al. (2020), and Liu et al. (2020).Source: §2 · p. 2

Cascaded ST

• Cascaded ST addresses error propagation with synthetic ASR-related errors and perturbations, while post-processing targets flaws such as segmentation, punctuation, and disfluencies.The cited post-processing examples include re-segmentation, punctuation restoration, and disfluency detection.Source: §2 · p. 2

Cross-Lingual Techniques

• Multilingual techniques also seek shared features across different representations, but do not address the audio-text difficulties; Chimera’s memory categories and objectives are described as simpler than Zhu et al. (2020).The cited multilingual approaches include interlingua models and cosine-based losses.Source: §2 · p. 2

3.1 Speech Translation Overview

• Chimera represents ST data as audio, transcript, and translation triplets and can also use parallel MT corpora.The MT corpus consists of source- and target-text sequences.Source: §3.1 · p. 2

3.2 Chimera Architecture

• Speech and text inputs are encoded into contextual features, projected into fixed-length semantic memories, and passed to a Transformer decoder.The projection uses trainable input-dependent memory queries across shared attentional layers; the decoder accepts memories from either modality.Source: §3.2 · pp. 3–4

3.3 Training Objectives

• ST negative log-likelihood and parallel-text translation training contribute to a weighted combination of the training losses.Parallel data can come from an MT dataset or transcript-translation pairs.Source: §3.3 · p. 4
• Bi-modal contrastive training aligns paired speech and text semantic memories by bringing matched elements closer while maintaining diversity among memories.The contrastive loss compares each text memory with speech memories, and vice versa.Source: §3.3 · p. 4

4.1 Dataset and Preprocessing

• MuST-C covers English-to-eight-language TED speech translation, while Augmented LibriSpeech supplies English-speech/French-text data and MT corpora include WMT, OPUS100, and OpenSubtitles.WMT 2014 has at least 4 million sentence pairs per language pair; OpenSubtitles has 22 million EN-DE pairs.Source: §4.1 · pp. 4–5

4.3 Benchmark Experiments

• Chimera achieves state-of-the-art results across the reported language pairs, and outperforms decoder pretraining and knowledge distillation under the same data condition.The EN-DE result uses WMT14 plus OpenSubtitles; the reported improvement on EN-PT is smaller than on EN-DE and EN-FR, which the authors attribute to OPUS100–MuST-C data discrepancy.Source: Table 1 · p. 5, Table 2 · p. 5, §4.3 · p. 7

4.2 Model Configuration

• Chimera is pretrained on MT and then fine-tuned on ST, MT, and a bimodal contrastive task; the main configuration uses 64 memory queries, while ablations use Mem-16.The shared Transformer encoder and decoder each have six layers.Source: §4.3 · p. 6, §4.2 · p. 6

4.3 Benchmark Experiments

• Without external MT pretraining, Chimera remains competitive but does not reach state-of-the-art performance.The benchmark setup pretrains on MT data before fine-tuning on ST and auxiliary tasks.Source: §4.3 · p. 6

4.4 Ablation Studies and Visualizations

• Increasing the amount of additional MT data significantly improves EN-DE MuST-C BLEU; the authors attribute weaker EN-PT results partly to OPUS100’s domain discrepancy.OPUS100 contains many informal, repetitive movie subtitles, unlike MuST-C public speeches.Source: Figure 3 · p. 7, §4.4 · p. 8
• Removing either auxiliary task significantly reduces BLEU, and the authors conclude that MT training and bimodal contrastive alignment work together to preserve and share MT knowledge.The reported one-tailed test comparing rows 1 and 2 gives p=0.020.Source: §4.4 · pp. 7–8
• PCA shows speech and transcript memories clustering in shared regions, while one memory’s subspace groups questions, future-tense sentences, and copular sentences.The visualized pairs remain close across modalities.Source: §4.4 · pp. 8–9, Figure 5 · p. 8
• Attention aligns matching audio and text tokens near the diagonal, while text periods align with audio pauses, which the authors interpret as evidence of semantic rather than positional memories.Source: Figure 6 · p. 9, §4.4 · p. 9

• Chimera achieves new state-of-the-art performance on the MuST-C dataset across all 8 languages.Source: §5 · p. 9
• Chimera learns a text-speech shared semantic memory network to bridge representation gaps and share learned knowledge across MT and ST.Source: §5 · p. 9
• MT and ST workflows remain only partially shared, requiring adaptation to ST during fine-tuning and leaving tighter computation-graph coupling for future work.Source: §5 · p. 9