Source-linked AI summary
S3-S2ST: Semantically Supervised Streaming Speech-to-Speech Translation
Yuxiang Zhao, Zhiyun Fan, Yu Pu, Yanjie An, Yushen Chen, Wenrui Liang, Baiji Liu, Yunchong Xiao, Zhaoci Liu, Keqi Deng, Kai Yu, Zhifu Gao, Xiangang Li, Xie Chen
TL;DR
Streaming speech-to-speech translation must decide when partial input is sufficient for accurate translation. S3-S2ST learns commitment from semantically aligned speech pairs and adjusts a threshold on a fixed input grid; it is competitive on FLEURS and leads evaluated streaming systems in translation quality on the internal 9-pair test set.
Problem
Streaming speech-to-speech translation requires deciding how much partial-speech context to wait for, balancing early responses against mistranslation and longer waits against interruption.
Source: §1 · p. 1Method
S3-S2ST turns utterance-level speech pairs into incremental examples that supervise translation and commitment, with a tunable threshold and playback adaptation on a fixed 640 ms input grid.
Source: §1 · p. 1, §3.3 · p. 3Results
S3-S2ST is competitive on FLEURS and leads evaluated streaming baselines in translation quality on the internal 9-pair test set.
Source: §1 · p. 2, §5.1 · p. 7Takeaways & Limitations
RealSI demonstrates quality–latency tradeoffs from a single checkpoint, with playback adaptation using buffered speech to wait for context.
Source: §1 · p. 1, §7 · p. 8Evaluation covers English and Chinese; broader multilingual transfer and long continuous conversations remain to be assessed, and the internal 9-pair set is not publicly released.
Source: Limitations · p. 8
Abstract
from arXiv · showhide
Streaming speech-to-speech translation requires timely decisions about when partial input is sufficient for accurate translation. We present S3-S2ST, a model that learns to translate and speak from semantically aligned streaming supervision. By converting utterance-level speech pairs into incremental training examples, we teach a learned policy when to commit translations. Its adjustable threshold controls the quality--latency tradeoff on a fixed 640\,ms input grid. S3-S2ST delivers competitive English--Chinese translation on FLEURS and leads the evaluated streaming systems in translation quality on the internal 9-pair test set. On RealSI, adapting the threshold to buffered speech further improves quality over a low fixed threshold with little change in first-audio latency. We plan to release the model and code.
1 Introduction
Streaming speech-to-speech translation must balance response delay against the risk of translating from insufficient context, especially because spoken output cannot be silently revised. S3-S2ST learns commitment from semantically aligned speech pairs and adjusts waiting behavior without changing its 640 ms input grid.
1 Introduction
- Streaming translation must decide how much context to wait for: early responses risk mistranslation, while longer waits interrupt interaction.Unlike displayed text, speech already heard cannot be silently revised.Source: §1 · p. 1
- The model converts utterance-level speech pairs into supervision for what to translate and when to speak, combining chunkwise encoding with continuous Thinker–Talker conditioning.It connects commitment timing to continuous speech generation in a single streaming model.Source: §1 · p. 1
- Fixed and playback-adaptive thresholds adjust commitment on an unchanged 640 ms input grid, providing quality–latency control without retraining.Playback adaptation uses buffered speech to wait for useful context without delaying the initial commitment.Source: §1 · pp. 1–2
- S3-S2ST is competitive on FLEURS and leads evaluated streaming baselines in translation quality on the internal 9-pair test set.The paper also reports favorable quality–latency results on RealSI.Source: §1 · p. 2
2 Related Work
Prior work coordinates streaming translation through fixed schedules, learned alignment or segmentation, and systems that learn from aligned speech streams. S3-S2ST distinguishes its latency control by varying a READ/WRITE threshold within an utterance, independently of input granularity and model weights.
2 Related Work
- Fluent speech generation alone does not determine when incomplete source speech provides enough context to translate.Prior work includes direct speech-unit prediction, text-guided speech generation, expressive and streaming systems, and time-aware dubbing.Source: §2 · p. 2
- Streaming policies range from fixed Wait-k schedules and learned segmentation or monotonic alignment to boundary-aware inference and aligned-stream supervision.Hibiki-Zero removes word-level alignment supervision and uses reinforcement learning to optimize latency while preserving translation quality.Source: §2 · p. 2
- SimulS2ST-Omni varies input chunk duration, while DIAL-S2ST adjusts a LoRA adapter; S3-S2ST instead varies its explicit READ/WRITE threshold within an utterance.The threshold can change with the playback buffer without changing model weights or input granularity.Source: §2 · p. 2
- Evaluation protocols cover semantic, acoustic, and temporal speech quality as well as simultaneous and online processing delay.These measures assess translation quality alongside the delay experienced by listeners.Source: §2 · p. 3
3 Method
S3-S2ST combines incremental source and text context with a Thinker that predicts text and READ/WRITE decisions, and a Talker that generates codec tokens for speech rendering. Semantic boundaries in paired transcripts supervise when to read and write on a fixed 640 ms grid; joint streaming training updates the source-to-codec model while keeping the waveform renderer frozen.
3.1 Model Interfaces
- The Thinker incrementally generates target text, while the Talker generates codec tokens from Thinker states for a frozen flow-matching decoder and vocoder to render as speech.The streaming context interleaves source-audio and translated-text spans; the offline model receives the complete source before translation.Source: §3.1 · p. 3, Figure 2 · p. 4
3.2 640 ms Chunkwise Encoder
- The frontend processes 640 ms chunks of 16 encoder states, and a block-causal adaptor reduces each chunk to eight positions.It permits within-chunk right context and 160 ms of lookahead at the first encoder layer; later layers cannot extend the future-context budget.Source: §3.2 · p. 3
3.3 Predicting When to Translate
- At each complete acoustic interval, an MLP predicts WRITE probability from the final normalized Thinker state, trained with equal-class-weight READ/WRITE cross-entropy.Inference writes when the probability reaches the threshold; end-of-input flushes any residual span, and raising the threshold requires stronger evidence without updating weights.Source: §3.3 · p. 3
- Playback adaptation sets the threshold between 0.1 and 0.99: an empty buffer restores 0.1, while playable speech raises it so the model can gather more source context.The rule changes inference decisions without additional training, and end-of-input still forces a flush.Source: §3.3 · p. 3
3.4 Eight-Layer Hidden-State Fusion
- The Talker is conditioned by a learned channelwise fusion of eight Thinker layers, while text and gate heads continue to use the final Thinker state.Fusion weights are learned jointly with the conditioning interface and shared across positions.Source: §3.4 · p. 3
3.5 Training Data and Supervision
- Paired transcripts are divided into aligned semantic units, whose forced-aligned endpoints are quantized to the 640 ms grid before READ/WRITE supervision is constructed.Each unit supervises READ at internal grid points and WRITE at its boundary, then target text and codec segments; unit lengths vary, but the observation grid remains fixed. The annotator uses complete transcript pairs, while inference predicts commitment from received source prefixes.Source: §3.5 · pp. 3–4
3.6 Speech Generation and Training
- After separate text and speech adaptation, joint streaming training updates the encoder, adaptor, Thinker, gate, fusion, projection, and Talker for source-to-codec optimization.The waveform renderer remains frozen; end-to-end training refers to source-to-codec optimization.Source: §3.6 · p. 4
4 Experimental Setup
The experiments use staged training to build a streaming speech-to-speech model, then evaluate translation, speech quality, and latency across FLEURS, an internal test set, and RealSI. The setup compares against offline and streaming baselines under specified inference and scoring protocols.
4.1 Training Data
- The streaming mixture contains approximately 13k hours of real source audio, balanced by direction and filtered for source and target ASR consistency, plus upsampled FLEURS speech.Offline speech adaptation uses approximately 300k hours of source speech; the final speech-training mixtures exclude the added 9-pair training supplement.Source: §4.1 · pp. 4–5
4.2 Training Stages
- Training proceeds through streaming S2TT adaptation, offline S2ST adaptation, and streaming S2ST adaptation, combining incremental translation and commitment learning with speech generation.The streaming stage includes Talker adaptation followed by joint source-to-codec optimization; the waveform renderer remains frozen.Source: §4.2 · p. 5
4.3 Inference
- At inference, the model reads 640 ms source blocks with 160 ms lookahead and uses a READ/WRITE gate; streaming speech is rendered in eight-token flow-matching chunks.The main tables use fixed τ = 0.5 and a playback-adaptive threshold; the offline model can use up to ten seconds of received speech.Source: §4.3 · p. 5
4.4 Evaluation Data and Metrics
- Evaluation covers English–Chinese FLEURS, an internal 9-pair test set, and RealSI; reported dataset scores average the two translation directions equally.Metrics include BLEU and COMET for text, ASR-based translation metrics, synthesis fidelity and naturalness measures, and computation-aware StartOffset.Source: §4.4 · pp. 5–6
4.5 Baselines
- The baselines include offline and streaming speech systems, text-only IWSLT for text metrics, and Hy-SimulMT with its official Qwen3-TTS (0.6B) backend.Hy-SimulMT is evaluated with both native chunk settings on FLEURS and 9-pair, and on RealSI.Source: §4.5 · pp. 6–7
5 Results
S3-S2ST performs strongly on the tested translation benchmarks, with the strongest reported translation results on the internal 9-pair set. On RealSI, its playback-adaptive policy improves the quality–latency operating point relative to fixed-threshold and baseline systems.
5.1 Translation and Speech Quality
- On 9-pair, S3-S2ST achieves the highest BLEU and COMET for both generated text and transcribed speech, while no system leads every reported metric.On FLEURS, the offline model is competitive with Qwen3-Omni on translation and leads evaluated systems in ASR-COMET and speaker similarity; Hy-SimulMT leads the FLEURS translation metrics.Source: §5.1 · p. 7
5.2 Quality and Latency
- Fixed τ = 0.1 improves both ASR-COMET and computation-aware StartOffset over SimulS2ST-Omni m2–m5; m1 responds earlier with lower quality.The fixed-threshold sweep traces the quality–latency tradeoff without changing the checkpoint or the 640 ms input grid.Source: §5.2 · p. 7
- On RealSI, playback-adaptive decoding improves on fixed τ = 0.1 at essentially unchanged onset and approaches the quality of fixed τ = 0.2 with an earlier response.The playback-adaptive point also achieves higher ASR-COMET and lower StartOffset than both Hy-SimulMT chunk settings.Source: Figure 3 · p. 7, §5.2 · p. 7
6 Ablations and Analysis
The ablations compare frontend recipes, commitment policies, playback adaptation, and speech-conditioning interfaces on FLEURS. They report gains from learned or adaptive decisions and from intermediate-layer fusion under the specified controlled comparisons.
6.1 Encoder Adaptation
- The chunkwise frontend recipe achieves stronger FLEURS translation quality than the earlier block-causal recipe, although the comparison evaluates complete recipes with differing initialization and training settings.Its text COMET of 0.827 is from an S2TT-stage checkpoint, distinct from the final S2ST model’s 0.828 at the same threshold.Source: Table 3 · p. 7, §6.1 · pp. 7–8
6.2 Commitment Policy
- The MLP commitment policy improves translation quality over writing every eight source blocks while responding earlier; writing every six blocks yields earlier onset at greater quality cost.The comparison holds model weights, rendering, and speaker conditioning fixed across policies.Source: Table 4 · p. 8, §6.2 · p. 8
6.3 Playback-Adaptive Threshold
- Playback adaptation improves ASR-BLEU and ASR-COMET over fixed τ = 0.1 with little change in mean first-audio latency.Both policies choose the same first-WRITE input position; the reported 70 ms StartOffset difference reflects runtime variation, with quality gains from subsequent commitments.Source: Table 5 · p. 8, §6.3 · p. 8
6.4 Speech-Conditioning Interface
- Eight-layer fusion yields modest gains in ASR-COMET and lower WER than final-layer conditioning, supporting intermediate representations for speech conditioning.The compared branches share initialization sources, training data, optimization budgets, and final decoding configuration.Source: Table 6 · p. 8, §6.4 · p. 8
7 Conclusion
S3-S2ST learns commitment from semantically aligned trajectories and integrates streaming translation with continuous speech generation. The conclusion reports strong translation performance on FLEURS and the internal 9-pair set, plus a quality–latency tradeoff on RealSI.
7 Conclusion
- FLEURS and internal 9-pair results establish strong translation performance against evaluated streaming baselines, while RealSI demonstrates a quality–latency tradeoff from one checkpoint.FLEURS ablations support learned commitments, playback adaptation, and multi-layer speech conditioning.Source: §7 · p. 8
- A 640 ms chunkwise frontend retains pretrained convolution alignment, while weighted Thinker states continuously condition the Talker.The model learns when to commit from semantically aligned trajectories; playback adaptation uses buffered speech to gather context.Source: §7 · p. 8
- The authors plan to release the model and its training, inference, and data-construction pipeline.Source: §7 · p. 8
Limitations
The evaluation covers English and Chinese; broader multilingual transfer and long continuous conversations remain to be assessed.
Limitations
- English–Chinese evaluations do not establish broader multilingual transfer or performance in long continuous conversations.Exact-match audits do not rule out near-duplicate audio or overlap inherited from pretraining, and automatic estimators may have language-dependent biases.Source: Limitations · p. 8
A Training Data and Supervision
The appendix describes how utterance-level speech pairs are converted into validated incremental supervision. It covers semantic alignment, trajectory construction, training-data handling, and policy selection.
A.1 Shared Speech-Pair Construction
- Source-language recordings are paired with target translations and synthesized target speech; the two directions use real source recordings rather than reversing synthetic targets.Source: §A.1 · p. 11
- Semantic units are aligned to both waveforms, checked for transcript and timing consistency, then rounded up to the 640 ms observation grid without an added block delay.The alignment mixture allows an 80 ms boundary tolerance.Source: §A.1 · p. 11
A.2 Semantic Segmentation and Validation
- Target segments may express only information supported by the available source prefix; adjacent segments are merged when a translation would anticipate later evidence.A second pass revises paired spans while preserving exact transcript text, and deterministic checks validate monotonic boundaries and reconstruction.Source: §A.2 · p. 11
A.3 From Semantic Units to Streaming Trajectories
- Semantic boundaries become READ/WRITE trajectories: units sharing a quantized source endpoint are merged, target codec spans are recomputed, and endpoint decisions receive WRITE labels.Source: §A.3 · p. 11, §A.4 · p. 12
A.4 Stage-Specific Mixtures; A.5 Data Separation and Policy Selection
- Trajectory construction applies stage-specific duration, coverage, and ASR-consistency checks before packing; policy screening uses validation recordings, with test quality excluded from selection.Source: §A.4 · p. 12, §A.5 · p. 12
B Training Stages and Optimization
The appendix lays out staged adaptation and the implementation and scoring procedures used to obtain and evaluate the reported models.
B.1 Pretrained Initialization
- The pretrained initialization combines a Qwen3.5-4B Thinker and SenseVoice-initialized speech encoder with a CosyVoice-initialized Talker.The Thinker and encoder start from approximately 3M hours of S2TT data; the Talker uses approximately 300k hours of S2ST data.Source: §B.1 · pp. 12–13
B.2 Streaming S2TT Adaptation
- Streaming S2TT first adapts the chunkwise encoder and adaptor with utterance-level supervision, then trains the Thinker and READ/WRITE gate on semantic trajectories.The second substage freezes the encoder and adaptor; its checkpoint supplies the frontend, Thinker, and gate for streaming S2ST.Source: §B.2 · p. 13
B.3 Offline S2ST Adaptation
- Offline S2ST trains the Talker and speech interface while keeping the source frontend and Thinker frozen, using an initial segment and a continuation segment.The reported lineage uses 5,000 initial updates followed by 10,000 continuation updates.Source: §B.3 · p. 13
B.4 Streaming S2ST Adaptation
- Streaming S2ST first adapts the Talker and speech interface with translation components frozen, then jointly updates the encoder, Thinker, gate, and speech components.The reported streaming checkpoint is taken at 4,000 E2E updates; flow matching and the vocoder remain frozen.Source: §B.4 · p. 14
C.1 Chunkwise Frontend
- The chunkwise frontend uses 640 ms blocks and constrained lookahead; its first complete block requires 802.5 ms of received source audio.Total waveform lookahead is 162.5 ms beyond each non-final chunk endpoint, and this waiting time counts toward computation-aware StartOffset.Source: §C.1 · p. 14
C.2 Gate and Layer Fusion
- Layer fusion combines signed channelwise coefficients over intermediate residual states and the final normalized Thinker state, without softmax normalization or input-dependent routing.Zeroing the first seven layer vectors and setting the final vector to one recovers final-state conditioning.Source: §C.2 · p. 14
C.3 Streaming Trajectories
- Each WRITE opens an assistant turn, while the next source span opens a user/audio turn; READ/WRITE labels supervise the gate rather than adding language-model tokens.Codec supervision uses five delay positions and includes beginning and ending tokens for each WRITE.Source: §C.3 · p. 14
D.1 Rendering and Scoring
- The scoring procedure compares generated translation text with ASR transcripts of output speech, pooling substitution, deletion, and insertion counts within each direction.The reported WER column is the unweighted mean of English WER and Chinese CER, not an average of per-utterance error rates.Source: §D.1 · p. 15
D.2 Evaluation Data and Latency
The evaluation uses public FLEURS and RealSI data alongside an internal 9-pair test set, with latency measured from real-time source replay to first waveform packet emission. Baseline settings and reported parameter counts are specified to contextualize system comparisons.
D.2 Evaluation Data and Latency
- Evaluation used public English–Chinese FLEURS test data and an internal 9-pair evaluation with 996 examples per direction shared across systems; public RealSI sentence-level scores averaged equally by direction.Source: §D.2 · p. 15
- StartOffset runs from real-time source replay to the first nonempty waveform packet, including source waiting and model generation through vocoding.Model loading, warmup, networking, and device playback buffering are excluded; emitted waveforms are concatenated for quality scoring.Source: §D.2 · p. 15
D.3 Baseline Configurations
- Main-table baselines use specified operating modes and source-segment settings, while the RealSI curve additionally sweeps latency multipliers m1–m5.UniSS is included only in the quality comparison; full commands, implementation revisions, and output traces are retained for reproducibility.Source: §D.3 · p. 15
D.4 Parameter-Count Scope
- Reported backbone scales exclude auxiliary speech modules, while dagger-marked Seamless entries use official model-level sizes.The listed LLM backbones range from S3-S2ST's 4B to Qwen3-Omni's 30B MoE Thinker, with approximately 3B active parameters per token.Source: §D.4 · p. 15
E Ablation Protocols
The ablation protocols compare frontend recipes, READ/WRITE commitment policies, speech-conditioning fusion, and playback-adaptive thresholds. The playback-adaptive threshold yields a positive ASR-COMET gain on FLEURS, while commitment-policy intervals differ across evaluation sets.
E.1 Frontend Recipes
- The historical encoder comparison uses an earlier block-causal text model after 12,000 joint updates, while the chunkwise comparator is the 10,000-update text model used for initialization.The recipes differ in initialization sources, frontend training, and data schedules; the historical run uses 640 ms blocks and a fixed WRITE threshold of 0.5.Source: §E.1 · p. 16
E.2 Commitment Policy
- MLP minus fixed K = 8 has 95% ASR-COMET intervals of [−0.0054, 0.0018] on 9-pair and [0.0074, 0.0233] on FLEURS.The fixed-schedule and MLP policies use fixed speaker conditioning from the initial source prefix.Source: §E.2 · p. 16
E.3 Speech Conditioning
- The final-layer fusion control follows the final streaming Talker and joint-training stages, and compares quality rather than runtime.Both fusion branches are evaluated on all 1,591 FLEURS examples, with direction-level scores averaged equally.Source: §E.3 · p. 16
E.4 Playback-Adaptive Threshold
- ASR-COMET gain is 0.014 with a 95% interval of [0.009, 0.019] for the FLEURS playback-adaptive threshold comparison.The comparison uses 945 Chinese-to-English and 646 English-to-Chinese examples per policy, with paired-utterance bootstrap resampling within directions.Source: §E.4 · p. 16