Source-linked AI summary
DIAL-S2ST: Learning Latency Control for Simultaneous Speech-to-Speech Translation
Jeongsoo Choi, Junseok Ahn, Joon Son Chung
TL;DR
Simultaneous speech-to-speech translation needs to balance quality and latency, but multistream models do not provide a separate read/write policy for inference-time adjustment. DIAL-S2ST uses lookahead self-distillation and a single scale-adjustable adapter; on French-to-English Hibiki, brief source-only adaptation achieves a favorable quality-latency trade-off. The findings are scoped to the evaluated model and setting.
Problem
Multistream simultaneous speech-to-speech models couple translation content with emission timing, making inference-time latency control less straightforward.
Source: §1 · p. 1Method
DIAL-S2ST distills lookahead predictions into a causal student and trains one LoRA adapter across matched lookahead scales, masking large text losses to account for the imitation gap.
Source: §1 · pp. 1–2Results
With approximately 10 hours of source speech and 500 optimization steps, DIAL-S2ST achieves a more favorable quality-latency trade-off than inference-time logit biasing on French-to-English Hibiki.
Source: §1 · p. 2Takeaways & Limitations
A single adapter provides a broad range of inference-time latency operating points after brief source-only adaptation.
Source: §1 · p. 2, §5 · p. 4Evaluation focuses on French-to-English Hibiki and primarily utterance-level speech, with measurements relying on automatic metrics rather than human evaluation.
Source: Limitations · p. 5
Abstract
from arXiv · showhide
We propose DIAL-S2ST, a post-training method for learning inference-time latency control in simultaneous speech-to-speech translation models. Our approach reuses pretrained translation capabilities through self-distillation from lookahead predictions, while limiting the influence of large text distillation losses to account for the imitation gap. This provides supervision for earlier emission without paired target data or reward computation. To make the learned timing adjustable at inference, we jointly train a single LoRA adapter across scales with matched lookahead guidance. Experiments on French-to-English translation with Hibiki demonstrate that only 10 hours of source speech enable this adaptation, providing flexible latency control with a favorable quality-latency trade-off.
1 Introduction
Simultaneous speech-to-speech translation must balance translation quality against latency, but multistream models such as Hibiki lack a separate inference-time read/write policy. DIAL-S2ST learns adjustable emission timing through lookahead self-distillation, and brief source-only adaptation yields a favorable quality-latency trade-off.
1 Introduction
- In multistream models such as Hibiki, translated text and speech streams encode both content and timing, leaving no separate read/write policy to adjust after training.Reading more source speech adds context but delays output, creating a quality-latency trade-off.Source: §1 · p. 1
- Logit biasing yields an unfavorable quality-latency trade-off, while supervised fine-tuning on shifted alignments requires separate training for each timing point.The paper targets inference-time control within one model while maintaining a favorable trade-off.Source: §1 · p. 1
- DIAL-S2ST uses source lookahead self-distillation to encourage earlier emission, masks exceptionally large text losses, and trains one LoRA adapter across scales for inference-time control.The approach requires neither paired target data nor reward computation.Source: §1 · pp. 1–2
- With approximately 10 hours of source speech and 500 optimization steps, DIAL-S2ST achieves a more favorable quality-latency trade-off than inference-time logit biasing on French-to-English Hibiki.A single adapter supports a broad range of latency operating points at inference.Source: §1 · p. 2
2 Method
The method uses the coupling between text and speech timing in multistream translation to train earlier emission from source-only data. Lookahead guidance, loss controls, and matched adapter scales provide adjustable latency at inference.
2.1 Multistream speech-to-speech translation
- Text and speech are generated as aligned, time-synchronous streams, with PAD filling text frames until the next word and functioning as a wait token.Because content and timing share token predictions, translation timing is learned together with translation content.Source: §2.1 · p. 2
2.2 Lookahead on-policy self-distillation
- A frozen lookahead teacher guides a causal student on student-generated trajectories, encouraging the student to emit translation content earlier using source speech heard so far.The source-only dataset does not require paired target translations or reward computation.Source: §2.2 · p. 2
- Speech predictions are regularized toward the no-lookahead teacher, while text KL terms above a fixed threshold are masked to limit large discrepancies caused by the imitation gap.The teacher can rely on source speech that the student has not yet heard.Source: §2.2 · pp. 2–3
2.3 Latency control with a single adapter
- Training pairs sampled lookahead with adapter scale, so inference-time latency is controlled by adjusting that scale; larger scales yield earlier output and zero scale recovers the pretrained model.The pairing teaches how emission timing should vary across scales.Source: §2.3 · p. 3
3 Experimental Setup
The experiments evaluate French-to-English speech translation using CVSS and compare DIAL-S2ST with four open-source systems. They specify quality, latency, and voice metrics alongside the adapter training setup.
3.1 Datasets
- The test set is CVSS, a widely used speech-to-speech translation dataset built on CoVoST 2 with Common Voice source speech; unless otherwise stated, post-training uses approximately 10 hours of out-of-domain FLEURS French source speech, and comparable-duration in-domain CVSS post-training is also evaluated.Source: §3.1 · p. 3
3.2 Evaluation metrics
- ASR-BLEU measures translation quality; LAAL and End Offset measure average translation delay and completion delay, while speaker-embedding similarity and UTMOS assess voice transfer and naturalness.ASR-BLEU compares Whisper-medium transcripts of generated speech with reference translations.Source: §3.2 · p. 3
3.3 Baselines
- The four baselines are StreamSpeech, Seamless, Hibiki, and Hibiki-Zero, with Hibiki serving as the base model.StreamSpeech uses input chunk size for latency control; Seamless uses a decision threshold, while Hibiki learns timing from word-aligned data and Hibiki-Zero optimizes the quality-latency trade-off through reinforcement learning.Source: §3.3 · p. 3
3.4 Implementation details
- The rank-16 LoRA adapter covers every Temporal Transformer linear layer and has 16 M trainable parameters; training runs for 500 steps with 16 rollouts per step.Training uses AdamW, a 5 × 10−5 learning rate, and linear warm-up; three-seed adapters are averaged with uniform model soups.Source: §3.4 · p. 3
4 Experimental Results
DIAL-S2ST provides broad inference-time latency adjustment with a competitive quality-latency trade-off, outperforming StreamSpeech in translation quality across comparable latencies. Ablations support student-generated trajectory distillation and identify effects of the KL design choices.
4.1 Main results
- Across broadly comparable latencies, DIAL-S2ST achieves substantially higher translation quality than StreamSpeech; logit biasing reduces latency at a sharp quality cost, while DIAL-S2ST spans a broad range with a competitive Pareto frontier.Figure 2 distinguishes connected points for different latency settings of one model from isolated points for distinct models.Source: Figure 2 · p. 4, §4.1 · p. 4
4.2 Ablations
- DIAL-S2ST achieves the highest ASR-BLEU at every matched-latency operating point, with its advantage over off-policy distillation increasing at lower latency.The off-policy variant uses lookahead teacher rollouts instead of student rollouts while retaining the same distillation objective.Source: Table 1 · p. 4, §4.2 · p. 4
- Masking large text KL terms improves translation quality at similar latency; removing acoustic KL lowers speaker similarity, and removing both semantic and acoustic KL further degrades translation quality and voice transfer..Source: Table 2 · p. 4, §4.2 · p. 4
- The full text distribution achieves comparable or better quality-latency trade-offs than grouped text KL, showing latency control can be learned without explicitly separating timing from lexical predictions.Source: Table 2 · p. 4, §4.2 · p. 4
- Reverse KL in all distillation terms causes silence collapse, while out-of-domain FLEURS training performs comparably to in-domain CVSS training.Source: Table 2 · p. 4, §4.2 · p. 4
5 Conclusion
DIAL-S2ST learns inference-time latency control for pretrained multistream simultaneous speech-to-speech translation models. The French-to-English Hibiki experiments show that a small amount of source speech can make learned emission timing adjustable at inference.
5 Conclusion
- DIAL-S2ST uses lookahead self-distillation to adapt emission timing in pretrained multistream models, and jointly trains across adapter scales so one adapter controls timing at inference.French-to-English Hibiki experiments show that a small amount of source speech suffices for this adaptation.Source: §5 · p. 4
Limitations
The evaluation focuses on French-to-English Hibiki and primarily uses utterance-level speech; the authors identify other models, language pairs, long-form translation, and human evaluation as future work.
Limitations
- The study focuses on French-to-English Hibiki and primarily utterance-level speech; extension to other models, language pairs, and long-form translation, plus human evaluation, remains future work.Source: Limitations · p. 5
Ethical Considerations
The work uses public datasets and pretrained models without additional human-subject data collection or user studies, and highlights responsible-use safeguards for deployment.
Ethical Considerations
- Experiments use publicly available FLEURS, CVSS, and Hibiki datasets and pretrained models, with no additional human-subject data collection or user studies.Source: Ethical Considerations · p. 5
- Generated speech may contain translation errors or be misused for voice impersonation in practical deployment.Source: Ethical Considerations · p. 5
- The authors encourage license compliance, speaker consent, disclosure of machine-generated translations, and human oversight in sensitive settings.Source: Ethical Considerations · p. 5