Source-linked AI summary
SimulU: Training-free Policy for Long-form Simultaneous Speech-to-Speech Translation
Amirbek Djanibekov, Luisa Bentivogli, Matteo Negri, Sara Papi
TL;DR
SimulS2S research commonly relies on costly training and short, pre-segmented inputs, limiting long-form continuous-speech evaluation. SimulU uses cross-attention in pretrained end-to-end models to manage history and output selection without retraining, and achieves strong quality-latency performance across eight MuST-C language pairs.
Problem
Existing SimulS2S systems often require resource-intensive training and evaluate short, pre-segmented utterances rather than continuous long-form speech.
Method
SimulU uses cross-attention in a pretrained end-to-end model to manage input and output history and select partial speech outputs without additional training or adaptation.
Results
SimulU consistently achieves the highest ASR-BLEU across six language directions and remains competitive on the other two, with 1–2 seconds start offset in most cases.
Takeaways & Limitations
SimulU offers a practical and competitive training-free approach to long-form simultaneous end-to-end speech translation.
Takeaways & Limitations
The reported acceptability limits for ear-voice span are approximately 2 seconds and vary with conditions and language pairs.
Abstract
from arXiv · showhide
Simultaneous speech-to-speech translation (SimulS2S) is essential for real-time multilingual communication, with increasing integration into meeting and streaming platforms. Despite this, SimulS2S remains underexplored in research, where current solutions often rely on resource-intensive training procedures and operate on short-form, pre-segmented utterances, failing to generalize to continuous speech. To bridge this gap, we propose SimulU, the first training-free policy for long-form SimulS2S. SimulU adopts history management and speech output selection strategies that exploit cross-attention in pre-trained end-to-end models to regulate both input history and output generation. Evaluations on MuST-C across 8 languages show that SimulU achieves a better or comparable quality-latency trade-off against strong cascaded models. By eliminating the need for ad-hoc training, SimulU offers a promising path to end-to-end SimulS2S in realistic, long-form scenarios.
1. Introduction
SimulS2S requires incremental translation decisions for continuous speech, but existing systems commonly depend on costly training, cascaded components, and short, pre-segmented inputs. SimulU addresses these gaps as a training-free policy for long-form end-to-end speech-to-speech translation.
- Simultaneous translation must balance reading new input with writing output from partial information in real time.
- Existing SimulS2S policies often require complex training pipelines, reinforcement learning, and large speech datasets with synthetic alignments.
- Cascaded systems compound errors, discard non-linguistic speech information, and incur latency because each component processes sequentially.
- Most prior evaluations use short, manually pre-segmented audio chunks, constraining systems to predefined boundaries rather than continuous speech.
- SimulU is proposed as the first training-free policy for long-form end-to-end speech-to-speech translation, using cross-attention to regulate history and output generation.
2. Methodology
SimulU repurposes cross-attention in a pretrained end-to-end model to manage speech and text history while selecting when to emit translated speech. Its workflow supports incremental long-form generation without additional training or adaptation.
- SimulU uses cross-attention scores to perform history management and speech output selection for direct simultaneous speech-to-speech translation.
- The policy repurposes offline pretrained models for simultaneous generation without additional training or adaptation.
- The workflow acquires incoming audio chunks incrementally, adds them to speech history, and generates intermediate textual hypotheses.
- Stable hypothesis selection uses speech-text cross-attention to emit tokens once aligned with an input unit.
- SimulU preserves a fixed word history and removes corresponding audio frames so retained speech and text remain aligned.
- The intermediate text is passed to text-to-unit generation and then to a vocoder for output speech synthesis.
3. Experimental Settings
The experiments evaluate SimulU on continuous MuST-C TED Talks across eight English-to-target language directions using quality and latency measures. It is compared with strong streaming cascades built from alternative policies and TTS systems.
- Dataset and evaluation: Evaluation covers English-to-Dutch, French, German, Italian, Portuguese, Russian, Romanian, and Spanish on complete MuST-C dev and test TED Talks.Average durations are 948 seconds for development and 650 seconds for testing.
- Dataset and evaluation: Quality is measured with ASR-BLEU on transcripts of translated speech, while latency is measured with SimulEval.Canary is used to transcribe long output speech before BLEU computation.
- Models: The SeamlessM4T backbone is a pretrained multilingual end-to-end model containing speech-to-text, text-to-unit, and vocoder components.Its architecture includes approximately 1B total parameters and a multilingual unit vocoder.
- Configuration: Preliminary en-de experiments select a word history of 10 because it performs better than or on par with alternatives while remaining relatively short.
- Baselines: SimulU is compared with StreamAtt and LocalAgreement cascades paired with Seam.TTS or XTTS-v2.StreamAtt controls latency through cut-off frames, whereas LocalAgreement uses speech segment size.
4. Results
SimulU delivers the strongest overall quality across the evaluated languages while maintaining mostly 1–2-second latency, and it also provides lower or more stable end-offset latency in several comparisons.
- SimulU consistently achieves the highest ASR-BLEU across six language directions and remains competitive on Russian and Dutch.
- At the same latency, SimulU and StreamAtt+XTTS-v2 outperform LA-based systems by at least 4–5 ASR-BLEU points.
- Seam.TTS cascades obtain ASR-BLEU scores of 5–10 with StreamAtt and 10–15 with LA, indicating severe degradation under partial-sentence conditioning.
- SimulU achieves lower end-offset latency than the strongest cascade for languages beyond German and French, with smaller standard deviation indicating more stable latency.
- SimulU achieves the best overall quality across the analyzed languages while maintaining latency between 1 and 2 seconds.
5. Conclusions
The paper presents SimulU as a training-free policy for long-form simultaneous speech-to-speech translation that uses pre-trained models’ internal cross-attention to regulate input history and output generation. Across eight language pairs, it performs strongly against state-of-the-art cascades without additional training, supporting its practicality for real-world simultaneous translation.
- SimulU is the first training-free long-form simultaneous speech-to-speech translation policy to dynamically regulate input history and output generation using cross-attention.
- Across eight language pairs, SimulU yields strong results against strong cascade systems without requiring additional training.