PaperlayerView exact revision on arXiv

Source-linked AI summary

StreamUni: Achieving Streaming Speech Translation with a Unified Large Speech-Language Model

Shoutao Guo, Xiang Li, Mengge Liu, Wei Chen, Yang Feng

arXiv:2507.07803v2cs.CLcs.SDeess.AS

TL;DR

StreamST must balance latency and translation quality, but existing approaches often rely on segmented clips and face challenges in unified policy learning. StreamUni uses speech CoT in a unified LSLM to manage policies and translation, and reports state-of-the-art performance across multiple directions.

  • Problem

    Existing StreamST approaches commonly rely on short, segmented SimulST clips, while effective policy learning and truncation remain challenging for LSLMs.

    Source: Introduction · pp. 1–2

  • Method

    StreamUni uses speech CoT multi-stage outputs in a unified LSLM to handle generation and truncation policies and streaming translation, with streaming CoT training on partial speech inputs.

    Source: Introduction · p. 2, Model Framework · p. 3

  • Results

    StreamUni achieves state-of-the-art performance on StreamST tasks across multiple directions.

    Source: Introduction · p. 2, Conclusion · p. 8

  • Takeaways & Limitations

    The framework coordinates segmentation, policy decisions, and translation through multi-stage outputs, with a unified training strategy for the subtasks.

    Source: Introduction · p. 2

Abstract

from arXiv · show

Streaming speech translation (StreamST) requires determining appropriate timing, known as policy, to generate translations while continuously receiving source speech inputs, balancing low latency with high translation quality. However, existing StreamST methods typically operate on sentence-level speech segments, referred to as simultaneous speech translation (SimulST). In practice, they require collaboration with segmentation models to accomplish StreamST, where the truncated speech segments constrain SimulST models to make policy decisions and generate translations based on limited contextual information. Moreover, SimulST models struggle to learn effective policies due to the complexity of speech inputs and cross-lingual generation. To address these challenges, we propose StreamUni, which achieves StreamST through a unified Large Speech-Language Model (LSLM). Specifically, StreamUni incorporates speech Chain-of-Thought (CoT) in guiding the LSLM to generate multi-stage outputs. Leveraging these multi-stage outputs, StreamUni simultaneously accomplishes speech segmentation, policy decision, and translation generation, completing StreamST without requiring massive policy-specific training. Additionally, we propose a streaming CoT training method that enhances low-latency policy decisions and generation capabilities using limited CoT data. Experiments demonstrate that our approach achieves state-of-the-art performance on StreamST tasks.

Introduction

StreamST must balance translation quality and latency while managing long speech streams, but existing approaches rely on short segmented clips and face policy-learning challenges. StreamUni uses speech CoT in a unified LSLM for policy decisions, truncation, and translation, with state-of-the-art results reported across multiple directions.

Introduction

  • StreamST needs generation and truncation policies to balance latency and translation quality over long streams.Generation timing governs when translated words are emitted; truncation removes historical speech and translations to focus on recent inputs and manage efficiency.Source: Introduction · p. 1
  • Existing methods rely on short, upstream-segmented SimulST clips, while LSLM approaches can require extensive policy-specific data and still leave truncation underexplored.The cascaded setup constrains policy decisions and translation quality; separate optimization of StreamST subtasks also requires substantial training data.Source: Introduction · pp. 1–2
  • StreamUni uses speech CoT multi-stage outputs to coordinate generation policy, truncation, and streaming translation in one LSLM.It detects effective speech chunks from intermediate transcriptions, compares current and historical transcriptions for truncation timing, and maintains truncation integrity by completing corresponding translations before discarding segments.Source: Introduction · p. 2
  • StreamUni achieves state-of-the-art performance on StreamST tasks across multiple directions.The proposed streaming CoT training scheme optimizes multi-stage outputs using partial speech inputs.Source: Introduction · p. 2

Background

StreamST generates translations as speech chunks arrive, using generation and truncation policies to manage timing and long-running input. Speech CoT describes transcription followed by translation.

Background

  • StreamST progressively generates translation words as speech chunks arrive, with each generation point determined by the number of chunks received.Speech chunks are typically around 320ms or 640ms.Source: Background · p. 2
  • For long streams, truncation policy sets speech and translation segment endpoints so the model can discard history and work from the segments that remain.The segmentation policy defines paired endpoints for speech and target translation.Source: Background · p. 2
  • StreamST requires both truncation and generation policies to guide translation generation.Figure 1 depicts the StreamUni framework and illustrates its truncation and generation policies.Source: Background · p. 3, Figure 1 · p. 3
  • Speech translation CoT proceeds from generating a transcription to producing the final translation.The passage presents this as the sequence used for speech translation.Source: Background · p. 3

Method

StreamUni uses speech Chain-of-Thought to unify transcription, truncation and generation-policy decisions, and translation generation in an LSLM. Its streaming CoT training uses sampled partial speech inputs to improve low-latency policy decisions and generation.

Model Framework

  • StreamUni encodes incoming speech, generates real-time transcriptions, and uses them to choose between truncation and generation-policy decisions within a unified LSLM.At truncation moments it completes the translation for the segment before discarding corresponding speech and translation; otherwise it selects how many translation words to output.Source: Model Framework · p. 3, Method · p. 3
  • Truncation preserves semantic alignment by avoiding untranslated content in discarded speech and translated content in retained speech; it triggers after repeated unchanged transcripts or a completed sentence followed by a new one.The framework selects prolonged silence or sentence completion as truncation timing, then outputs the complete translation for the truncated input segment.Source: Model Framework · pp. 3–4
  • At non-truncation times, StreamUni generates translation only when transcription detects input text and delays output by k words to provide context.The generation policy uses the current transcription and the count of already-output translation words to determine how many words can be emitted.Source: Model Framework · p. 4

Streaming CoT Training

  • Streaming CoT data are constructed from non-streaming speech, transcription, and translation triplets by sampling partial streams and aligning their corresponding transcripts.The training objective predicts full translations from partial speech and transcription.Source: Streaming CoT Training · p. 4
  • Uniformly sampled partial inputs train transcription prediction for policy decisions and full-translation prediction for generation; experiments combine this training with traditional non-streaming training for greater gains.Sampling avoids training on every possible speech input.Source: Streaming CoT Training · p. 4

Experiments

Experiments evaluate StreamUni on SimulST and StreamST. The reported results show optimal SimulST performance across datasets and superior StreamST performance.

Datasets

  • MuST-C En⇒De and En⇒Es support evaluation on both StreamST and SimulST, with En⇒De containing document-level and human-annotated sentence-level data.Source: Datasets · pp. 4–5

Main Results

  • StreamUni achieves optimal SimulST performance across datasets and latency settings, with further gains from its policy and streaming CoT training scheme.Source: Figure 2 · p. 5, Main Results · pp. 5–6

Datasets

  • CoVoST2 En⇒Zh and Fr⇒En are used to evaluate SimulST; the En⇒Zh data contains only sentence-level speech translation.Source: Datasets · p. 5

System Settings

  • Phi-4-Multimodal is the primary backbone, fine-tuned on four language directions; Qwen2.5-Omni is also tested for generalizability.Source: System Settings · p. 5
  • Phi4-Wait-k waits k−1 chunks before emitting one word per subsequent chunk, while Phi-4-VAD substitutes VAD for StreamUni’s truncation policy.Source: System Settings · p. 5

Evaluation

  • Evaluation measures latency with AL and LAAL and translation quality with SacreBLEU and COMET; StreamST uses StreamLAAL and Stream SacreBLEU.Source: Evaluation · p. 5

Main Results

  • StreamUni outperforms existing StreamST methods, using semantic alignments between speech and translation to determine truncation timing instead of VAD.Source: Figure 3 · p. 6, Main Results · p. 6

Analysis

The analyses find that streaming CoT training improves performance across latency settings, with the strongest gains requiring non-streaming data alongside streaming data. Evaluations also report strong performance across speech CoT stages and generalization to another LSLM, while document-level truncation performs competitively with human timing.

Ablation Study

  • Streaming CoT training enhances performance across latency settings, particularly at low latency, but maximum gains require combining it with non-streaming data.Figure 4 compares generation policies and training methods on CoVoST2 En⇒Zh; the text also notes that the mixed-data recipe is required for maximum gains.Source: Ablation Study · p. 6, Figure 4 · p. 6
  • On MuST-C En⇒De, increasing streaming-only CoT data does not improve performance; superior results across latency settings require both streaming and non-streaming CoT data.Figure 5 compares the proposed recipe using 50 hours of each data type with methods using varying durations of streaming data only.Source: Ablation Study · p. 6, Figure 5 · p. 6
  • On document-level MuST-C speech averaging over 10 minutes, the proposed truncation strategy scores slightly below human timing on SacreBLEU but above it on COMET.The evaluation disables generation policy and reports document-level metrics; Table 1 compares human-annotated and model-determined truncation timing.Source: Ablation Study · p. 7, Table 1 · p. 7

Speech CoT Argumentation

  • StreamUni’s approach achieves superior performance across all speech CoT stages, supporting policy decisions and streaming translation generation.The evaluation uses a low-latency CoVoST2 En⇒Zh dataset; Table 2 compares multiple CoT stages under different training configurations.Source: Speech CoT Argumentation · p. 7, Table 2 · p. 7

Extending to Other LSLMs

  • Phi-4-Multimodal consistently outperforms Qwen2.5-Omni-7B on ST and SimulST, and StreamUni leverages and scales with stronger LSLM capabilities.The comparison is presented in Table 3.Source: Extending to Other LSLMs · p. 7, Table 3 · p. 7

Related Work

Prior SimulST work has explored chunk-based and content-adaptive generation policies, but sentence-level speech clips do not directly address streams lasting tens of minutes. Unified LSLM approaches aim to handle StreamST subtasks in one model, yet face data-construction, transferability, and use-of-pretraining challenges.

Related Work

  • SimulST methods have used chunk counts and content-adaptive signals, including auxiliary ASR, integrate-and-fire, monotonic attention, transducers, and CTC, for generation decisions.These methods are described for sentence-level speech segments.Source: Related Work · p. 7
  • LSLM SimulST systems still need auxiliary models for complete StreamST, while unified approaches face policy-data construction, transferability, and foundation-model pretraining challenges.Prior unified methods include attention-based generation and segmentation decisions and policy-specific data for autoregressive policy prediction.Source: Related Work · pp. 7–8

Conclusion

StreamUni enables a unified LSLM to handle all StreamST subtasks cohesively, and the paper reports state-of-the-art performance across multiple directions.

Conclusion

  • StreamUni enables a unified LSLM to accomplish all StreamST subtasks cohesively, with state-of-the-art performance across multiple directions.The conclusion reports the method’s performance across multiple directions.Source: Conclusion · p. 8

Truncation Policy in StreamUni

StreamUni truncates historical speech and translations in real time to focus on recent inputs while avoiding additional inference costs. It selects source cutoffs at prolonged silence, sentence completion, or—if neither occurs—after 30 chunks, then aligns translation truncation to the source cutoff.

Truncation Policy in StreamUni

  • The truncation policy removes historical speech and translations to focus on recent inputs while avoiding additional inference costs.It avoids removing untranslated speech or retaining already-translated content in the remaining speech segment.Source: Truncation Policy in StreamUni · p. 10
  • Speech truncation is triggered by prolonged silence or sentence completion; if neither occurs before the processed speech exceeds 30 chunks, that moment becomes the cutoff.The cutoff is intended to correspond to a pause or completed sentence.Source: Truncation Policy in StreamUni · p. 10
  • After identifying the source cutoff, StreamUni outputs translations preceding it and discards them to align source and target truncation.The alignment is intended to preserve semantic consistency between discarded source and target content.Source: Truncation Policy in StreamUni · p. 10

Training and Evaluation Details

Training data uses streaming CoT for En⇒Zh and non-streaming data for other language pairs. StreamST evaluation aligns document-level translations with sentence-level references before measuring latency and translation quality.

Training and Evaluation Details

Loading 2507.07803v2…