PaperlayerView exact revision on arXiv

Source-linked AI summary

Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systems

Marco Gaido, Sara Papi, Mauro Cettolo, Matteo Negri, Luisa Bentivogli

arXiv:2512.17648v3cs.CL

TL;DR

StreamST evaluation lacks a unified framework spanning long-form inputs, both decoding paradigms, and interactive demonstration. simulstream combines those capabilities with detailed quality and latency logging and a real-time web interface. It became the official evaluation toolkit for IWSLT 2026, and its first direct comparison reports StreamAtt on Seamless ahead of sliding-window retranslation in latency and translation quality while eliminating flickering.

  • Problem

    Existing frameworks support only subsets of StreamST evaluation, leaving no unified solution for fair comparison across decoding paradigms, long-form inputs, and interactive demonstrations.

    Source: §1 · p. 2

  • Method

    simulstream is an open-source framework for long-form StreamST evaluation and demonstration across incremental and re-translation decoding, with token tracking and an interactive web interface.

    Source: §1 · p. 2

  • Results

    simulstream became the official evaluation toolkit for IWSLT 2026; its first direct comparison found StreamAtt on Seamless outperformed sliding-window retranslation in latency and translation quality while eliminating flickering.

    Source: §1 · p. 2, §3.2 · p. 6

  • Takeaways & Limitations

    The framework provides a paradigm-agnostic platform to systematically study, benchmark, and showcase modern StreamST approaches.

    Source: §4 · p. 6

Abstract

from arXiv · show

Streaming Speech-to-Text Translation (StreamST) requires producing translations concurrently with incoming speech under strict latency constraints, demanding models that balance low latency with high translation quality. Despite rapid progress, evaluation remains fragmented across existing frameworks, which make different assumptions about how systems operate - for example, whether they process continuous speech or short pre-segmented audio, and whether they support output revision (retranslation) or not (incremental). For instance, SimulEval, the most widely used framework, supports only incremental decoding, assumes short segmented inputs, and lacks a native support for system demonstrations. As a result, comparing systems fairly and consistently across studies remains challenging, with no unified solution for benchmarking and interactive demonstration. To address this gap, we introduce simulstream, the first open-source framework for StreamST evaluation and demonstration. It supports both incremental and re-translation decoding on long-form speech, provides fine-grained logging for quality and latency evaluation, and includes an interactive web interface for real-time visualization and comparison.

1 Introduction

StreamST systems must translate continuous speech using only partial information while balancing latency and translation quality. simulstream addresses fragmented evaluation with a unified open-source toolkit for both decoding paradigms, long-form audio, and interactive demonstration.

1 Introduction

  • StreamST requires translation to keep pace with incoming speech while systems balance low latency against high translation quality.Unlike offline translation, systems make decisions with only partial information available.Source: §1 · p. 1
  • Retranslation can revise earlier output but often flickers, whereas incremental decoding is append-only and offers a more stable experience.Retranslation is prevalent in industry, while research has largely focused on incremental methods.Source: §1 · p. 1
  • Existing frameworks leave a gap: SimulEval (Ma et al., 2020) supports only incremental decoding and short segments, while alternatives lack either demonstrations or an open evaluation backend.The paper also notes SimulEval is no longer actively maintained.Source: §1 · p. 2
  • simulstream unifies long-form evaluation across incremental and re-translation strategies, tracks emitted and deleted tokens, and offers a SimulEval proxy for reusing existing systems.The framework is released under the Apache 2.0 License.Source: §1 · p. 2
  • Its interactive web demonstration platform enables real-time visualization and system comparison.Source: §1 · p. 2
  • The unified infrastructure has been adopted as the official evaluation toolkit for the Simultaneous Translation track of the IWSLT 2026 Evaluation Campaign.Source: §1 · p. 2

2 The simulstream Tool

simulstream runs streaming speech processors over continuous audio and supports both demonstrations and evaluation. Its server-client architecture, extensible processors, and fine-grained logs underpin quality, latency, and computational-cost assessment.

2 The simulstream Tool

  • The WebSocket tool processes continuous, unbounded audio streams for real-time transcription and translation; short-form evaluation can be emulated by splitting audio into consecutive segments.Source: §2 · p. 2

2.1 Architecture

  • Clients send configuration and audio chunks to the server, which returns transcripts or translations and can log generated and deleted text, computational costs, and audio processed.A fixed-size processor pool controls concurrency; connections beyond capacity are rejected. The web client supports demonstrations, while WAV and direct-audio commands produce evaluation logs.Source: §2.1 · pp. 2–3

2.2 Speech Processors

  • Example processors include sliding-window retranslation with overlap deduplication, StreamAtt using cross-attention alignments, VAD chunking, and a SimulEval-agent wrapper.Custom processors can be added through the SpeechProcessor interface and must return incremental outputs, including any tokens to delete.Source: §2.2 · p. 3

2.3 Evaluation

  • Evaluation uses fine-grained logs to compute built-in BLEU and COMET quality scores, with resegmentation against sentence-level references; revised systems are scored on their final output.Source: §2.3 · p. 4
  • StreamLAAL measures latency from word emission times against uniformly distributed reference-word times; StreamLAALCA also accounts for model execution time.For retranslation, latency uses each word’s last update time, which may overestimate user-experienced latency while excluding intermediate hypotheses from quality scoring; normalized erasure and real-time factor provide additional statistics.Source: §2.3 · p. 4

3 Experiments

The experiments compare streaming strategies on long-form speech across datasets and models. Results show that the quality–latency trade-off depends on the model and decoding strategy, with no universally superior approach.

3.1 Setup

  • The comparison includes sliding-window retranslation, a VAD-wrapped sliding window, and incremental StreamAtt, with operating points varied through windows, thresholds, and frame cutoffs.Sliding windows advance by 2 seconds; StreamAtt uses 1-second speech chunks, and inference runs on one NVIDIA A40 40GB.Source: §3.1 · pp. 4–5
  • The authors also experimented with Seamless large v2 but obtained inferior results, and refer readers to Appendix A for further details about that comparison.Source: §3.1 · p. 4
  • The evaluation uses full TED talks from eight MuST-C language pairs and scientific MCIF presentations, rather than sentence-level audio.MuST-C talks last about 10 minutes; MCIF presentations last about 5–6 minutes and include varied recording quality, accents, and domain terminology.Source: §3.1 · p. 5

3.2 Results

  • Canary retranslation outperforms SeamlessM4T on MuST-C quality at similar latency, with less flickering and lower computational cost.The comparison shows analogous trends with and without the VAD wrapper.Source: §3.2 · p. 5
  • Adding the VAD wrapper reduces flickering roughly 8× and RTF 2–3×, but sharply lowers translation quality and narrows the latency–quality range.VAD operating points span 0.2 seconds in latency and less than 0.015 COMET.Source: §3.2 · pp. 5–6
  • No single decoding strategy is universally superior: Canary favors retranslation quality, while StreamAtt with Seamless yields the lowest latency.The same model-dependent pattern is reported on MCIF.Source: §3.2 · p. 6, Figure 3 · p. 6

4 Conclusions

Simulstream is an open-source framework for evaluating and demonstrating StreamST systems. It supports both decoding paradigms on long-form audio, with token tracking and a real-time interactive interface.

4 Conclusions

  • Simulstream supports incremental decoding and retranslation on long-form audio, tracking token emissions and deletions while evaluating quality, latency, and computational cost.The framework is designed for unified evaluation and demonstration of StreamST systems.Source: §4 · p. 6
  • Its interactive web interface visualizes and compares StreamST systems in real time.The authors present this as a bridge between research and practical deployment.Source: §4 · p. 6
  • Incremental decoding remains essential for cascaded speech-to-speech translation systems.The paper gives cascaded speech-to-speech systems as an application example.Source: §4 · p. 6

A Comparison with SeamlessM4T v2 Large

SeamlessM4T v1 medium outperforms v2 large on nearly all reported metrics, and the paper uses v1 medium for its main results.

A Comparison with SeamlessM4T v2 Large

B Numerical Results on MCIF

Table 3 presents MCIF numerical results spanning speech processors, language pairs, and models.

B Numerical Results on MCIF

Loading 2512.17648v3…