PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenSTBench: Beyond Semantic Evaluation for Speech Translation

Yanjie An, Yuxiang Zhao, Yichi Zhang, Qixi Zheng, Yujie Tu, Keqi Deng, Kai Yu, Xie Chen

arXiv:2605.30792v1eess.AScs.AI

TL;DR

Speech translation evaluation often separates translation, speech, and temporal assessment, complicating comprehensive comparison across heterogeneous systems. OpenSTBench unifies these dimensions across S2TT and S2ST in offline and streaming settings, showing substantial cross-dimensional ranking variation among representative systems.

  • Problem

    Existing protocols often evaluate translation, speech, and temporal quality separately, limiting comprehensive comparison of heterogeneous speech translation systems.

  • Method

    OpenSTBench uses a shared protocol to evaluate S2TT and S2ST systems across offline and streaming settings on translation, speech, and temporal quality.

  • Results

    Representative speech translation systems exhibit substantial variation in rankings across evaluation dimensions.

  • Takeaways & Limitations

    Speech translation systems should be compared and selected according to application priorities and cross-dimensional trade-offs rather than a single global ranking.

  • Takeaways & Limitations

    Experiments focus on EN↔ZH, automatic speech evaluators lack fully validated correspondence to human judgments, and some system comparisons are not strictly matched.

Abstract

from arXiv · show

Speech translation systems increasingly span speech-to-text translation (S2TT), speech-to-speech translation (S2ST), offline translation, and streaming generation, producing outputs that differ in modality, speech realization, and timing behavior. Existing evaluation practices assess important aspects such as translation quality, speech quality, and temporal quality, but these aspects are often evaluated under separate protocols, making it difficult to compare heterogeneous systems comprehensively. To address this gap, we present OpenSTBench, a unified multidimensional evaluation framework that organizes heterogeneous speech translation outputs into a shared evaluation format. OpenSTBench supports both S2TT and S2ST systems in offline and streaming settings, and jointly evaluates translation quality, speech quality, speaker preservation, emotion and paralinguistic fidelity, temporal consistency, and latency. Through experiments on representative speech translation systems, we show that systems with strong translation quality can still differ substantially in speech quality, as well as in temporal quality. OpenSTBench provides a reproducible protocol for analyzing these cross-dimensional differences and supporting application-oriented comparison of speech translation systems. The code and datasets are available at https://github.com/sjtuayj/OpenSTBench.

1 Introduction

Speech translation now spans heterogeneous offline and streaming modalities whose behavior cannot be characterized by translated text alone. OpenSTBench addresses this gap with a unified, reproducible framework for multidimensional evaluation and application-oriented comparison.

  • Speech translation outputs differ in naturalness, speaker characteristics, expressive information, acoustic events, duration structure, and response latency.
  • Translation quality remains essential, but S2ST and streaming ST evaluation must also assess speech realization, preserved speaker and expressive information, and timing.
  • Existing evaluation dimensions are usually applied under task-specific protocols, limiting comprehensive comparison across heterogeneous speech translation systems.
  • OpenSTBench unifies S2TT and S2ST evaluation across offline and streaming settings through shared records, a common evaluator interface, and a consistent output schema.It consolidates translation quality, speech quality, and temporal quality, selecting metrics according to available output modality and timing information.
  • Representative systems show substantial cross-dimensional ranking variation, motivating application-oriented selection based on priorities and trade-offs rather than a single global ranking.
  • The framework establishes a reproducible protocol that standardizes data organization, scoring, and aggregation across heterogeneous speech translation systems.

2 Related Work

Related work evaluates speech translation through separate translation-quality, speech-side, and timing-oriented protocols. OpenSTBench builds on these ingredients and SimulEval’s quality-latency evaluation while organizing them under a shared evaluation format.

  • BLEU and COMET traditionally measure speech translation quality against reference translations, especially for S2TT.
  • Speech-side evaluation increasingly includes naturalness, intelligibility, speaker similarity, and latency as speech translation expands to S2ST and streaming generation.
  • UTMOS predicts MOS-style speech naturalness and quality, while Resemblyzer and WavLM support embedding-based speaker similarity and emotion2vec provides affective representations.
  • AudioSet and DCASE study acoustic event detection, whereas event preservation in cross-lingual speech translation is not addressed by those benchmarks.
  • SimulEval standardizes online interaction and quality-latency measurement, and OpenSTBench extends this foundation with shared records, interfaces, and output schemas for unified evaluation.

Speech Quality

OpenSTBench represents heterogeneous S2TT and S2ST outputs in a shared evaluation record, enabling comparable speech-quality profiles alongside translation and temporal quality.

  • Speech Quality: OpenSTBench uses a shared evaluation record for heterogeneous S2TT and S2ST outputs.The framework is designed to represent outputs from both speech-to-text and speech-to-speech translation systems.
  • Speech Quality: The framework reports comparable profiles across translation quality, speech quality, and temporal quality.These dimensions are presented jointly rather than as isolated output representations.
  • Speech Quality: OpenSTBench defines evaluation dimensions and metrics according to the system types to which each metric group applies.The evaluation table specifies applicability through its Applies-to field.

3 OpenSTBench

OpenSTBench provides a unified, reproducible protocol for evaluating S2TT and S2ST systems in offline and streaming settings. It organizes translation, speech, and temporal quality under shared interfaces while supporting modular evaluation across heterogeneous outputs.

  • Framework workflow: OpenSTBench evaluates S2TT and S2ST systems in offline and streaming settings through a shared input format, evaluator interface, and output schema.The protocol supports reproducible scoring and modular extension to new datasets and evaluation components.
  • Translation quality: Translation quality uses sacreBLEU, chrF++, COMET, and BLEURT to capture lexical, character-level, and semantic correspondence with reference translations.This dimension applies to both S2TT and S2ST, using generated text or transcriptions when needed.
  • Speech quality: Speech quality measures naturalness, realization fidelity, speaker preservation, emotion preservation, and paralinguistic fidelity.It uses UTMOS, Whisper-based CER or WER, WavLM and Resemblyzer similarity, Emotion2Vec cosine similarity, emotion classification accuracy, and CLAP-based Event Content F1 and Event Timing F1.
  • Temporal quality: Temporal quality covers speech temporal consistency and streaming latency, using SLC0.2, SLC0.4, Start Offset, ATD, Custom ATD, and RTF.Higher SLC values indicate better temporal consistency, while latency scoring follows SimulEval and includes computation-aware variants when applicable.
  • Open-source reuse: The framework separates system outputs from evaluation modules so systems share one input schema while evaluators can be replaced or extended independently.This design enables comparison under a common reporting format across different output modalities and timing information.

4 Experimental Setup

The experiments compare representative speech translation systems across streaming and offline settings using modality- and timing-aware evaluation protocols. OpenSTBench combines direction-specific datasets, multidimensional metrics, and system-specific inference procedures for reproducible comparison.

  • Systems: Streaming evaluation covers three S2ST systems—Qwen3-LiveTranslate, Doubao AST 2.0, and GPT Realtime Translate—and Baidu Realtime ST as a streaming-input S2TT baseline.Baidu emits sentence-final text rather than fully incremental speech, so its latency is reported as sentence-final delay.
  • Systems: Offline evaluation covers SeamlessM4T-v2-Large and UniSS, which generate target speech and are evaluated on translation and speech-side dimensions except streaming latency.Their parameter counts are 2.3B and 1.5B, respectively; API-based streaming systems do not publicly disclose parameter counts.
  • Datasets: The evaluation uses MSLT for EN→ZH and ZH→EN translation quality, speech quality, temporal consistency, and latency, with up to 1,000 dev samples per direction.Separate datasets cover emotion and paralinguistic fidelity, while a 300-sample LibriTTS-based paired set supports same-language speaker preservation.
  • Metrics: All systems receive sacreBLEU, chrF++, COMET, and BLEURT evaluation, while target-speech systems additionally receive UTMOS, CER/WER, speaker, emotion, paralinguistic, and temporal metrics.Temporal quality uses Start Offset, ATD, Custom ATD, and RTF when applicable.
  • Inference protocol: Streaming systems use native online interfaces with incremental source audio, whereas offline S2ST systems process complete segments and report RTF instead of incremental delay metrics.Offline RTF is measured on a single NVIDIA GeForce RTX 3090 GPU using each model’s native inference configuration.

5 Results and Analysis

Results across translation, speech, and temporal dimensions show that no single system dominates every aspect of speech translation. Text-side quality, speech preservation, responsiveness, temporal consistency, and efficiency therefore require joint, application-oriented comparison.

  • Translation quality: Qwen3-LiveTranslate is consistently strongest across both directions and metric families, while Doubao AST 2.0 and UniSS are the strongest streaming and offline alternatives, respectively.GPT Realtime Translate and Baidu Realtime ST are weaker on text-side quality.
  • Speech quality: UTMOS and ASR-based CER/WER favor different systems, so perceived speech naturalness and accurate target-text realization should be evaluated separately.UniSS and Doubao AST 2.0 show consistently strong speaker preservation, while emotion preservation varies across complementary evaluators.
  • Speech quality: Paralinguistic fidelity remains weak across systems, with low absolute Event Content F1 and Event Timing F1 despite direction-dependent ranking differences.Preserving acoustic event content and timing remains a major challenge for current speech translation systems.
  • Temporal quality: Streaming delay metrics, temporal consistency, and offline RTF capture distinct properties: earlier output does not ensure consistency, and efficient offline systems can differ substantially in SLC.Responsiveness, duration preservation, and offline computation reflect different deployment-relevant constraints.
  • Cross-dimensional analysis: No single system dominates every dimension: strong text-side performance does not imply speech-side preservation, and low latency does not guarantee temporal consistency.These cross-dimensional differences motivate application-oriented comparison under a unified multidimensional protocol.

6 Conclusion

OpenSTBench is a unified, extensible framework for multidimensional speech translation evaluation across S2TT, S2ST, offline, and streaming settings. Its shared protocol enables comparison of heterogeneous outputs and motivates application-oriented evaluation based on cross-dimensional variation rather than a single global ranking.

  • OpenSTBench provides a unified framework for multidimensional evaluation across S2TT, S2ST, offline, and streaming speech translation settings.
  • Its shared protocol organizes translation quality, speech quality, and temporal quality for comparing heterogeneous system outputs.
  • Experiments reveal substantial cross-dimensional variation, motivating application-oriented comparison instead of a single global ranking.
  • OpenSTBench is presented as an open-source and extensible framework.

Limitations

OpenSTBench is limited by its current EN↔ZH scope, reliance on automatic evaluators, and asymmetries across streaming and offline systems. Future work targets broader language coverage, stronger human calibration, realistic interactive settings, and extensible open-source evaluation.

  • Empirical scope: OpenSTBench’s experiments focus on EN↔ZH, leaving broader empirical validation across additional language pairs for future work.The benchmark is intended to expand beyond its current language-pair coverage.
  • Metric validation: Automatic evaluators for speech quality and preservation require further validation against human judgments.Future work should strengthen calibration between these metrics and human assessments.
  • System comparability: Streaming S2ST, streaming S2TT, and offline S2ST differ in output form or generation regime, limiting strictly matched comparisons.The asymmetry complicates direct comparison across these system settings.
  • Future directions: Future work should extend the benchmark to more language pairs and realistic interactive settings while examining its evolving multidimensional comparison protocol.The framework is intended to remain useful as speech translation systems continue to evolve.
  • Reproducibility and extensibility: OpenSTBench will be released as an open-source Python package with a public repository supporting reproducible and extensible research.Users can organize system outputs, run multidimensional evaluators, and replace or extend components for new datasets, language pairs, and research goals.

Ethical Considerations · A Additional Results

OpenSTBench is intended for aggregate research evaluation, with explicit safeguards for speaker-related resources, dataset licensing, privacy, and human-subject considerations. The authors also report that AI assistants supported limited editorial tasks while authors verified the technical content and results.

  • Ethical Considerations: OpenSTBench should not be used for speaker verification, impersonation, surveillance, or other identity-sensitive deployments.Its speaker-preservation and voice-cloning resources are intended for aggregate system comparison, not individual speaker identification.
  • Ethical Considerations: The benchmark uses publicly available datasets, models, and evaluation tools under their released access conditions and licenses.The LibriTTS-based paired speaker set is restricted to research evaluation, with derivative artifacts governed by the original resources’ intended use.
  • Ethical Considerations: The released OpenSTBench package will document supported datasets, language directions, input/output formats, evaluation components, and intended research use.This documentation is part of the planned release package.
  • Ethical Considerations: The speech datasets may contain speaker-identifying voice information, but the authors neither infer real-world identities nor collect additional personal information.They also do not release private metadata beyond what the original datasets provide.
  • Ethical Considerations: No new human participants were recruited and no new human-subject data were collected, so participant instructions, recruitment, payment, and ethics-board approval were not applicable.The passage also states that the work introduces no new offensive content.
  • Ethical Considerations: AI assistants supported language polishing, checklist preparation, and minor editing suggestions, while the authors reviewed and verified all technical claims, experimental results, and final manuscript content.The reported AI-assistant use was limited to editorial support.

A.1 Full Speaker Similarity Anchor Results … C Radar Plot Normalization Ranges

The paper reports speaker-similarity anchor analyses, defines a multidimensional metric suite, and specifies fixed radar-plot normalization ranges. Anchor choice substantially affects cross-lingual speaker-similarity scores, while fixed ranges support comparability across systems.

  • A.1 Full Speaker Similarity Anchor Results: Anchor choice substantially affects measured speaker similarity, especially for WavLM, because comparisons vary in language and natural-versus-synthesized reference conditions.Both source-anchor and target-anchor scores are reported by translation direction and system.
  • A.1 Full Speaker Similarity Anchor Results: Same-language target-side anchors are used for the main speaker-preservation evaluation, but cross-lingual automatic similarity scores require careful interpretation.In EN→ZH, the target anchor is synthesized Chinese speech; in ZH→EN, it is original LibriTTS English speech.
  • B.1 Translation Quality: The evaluation covers translation quality with sacreBLEU, chrF++, COMET, and BLEURT, where higher values indicate better translation quality.These metrics are reported as part of the translation-quality evaluation.
  • B Metric Definitions: The metric suite jointly represents translation, speech, speaker, emotion, paralinguistic, temporal, and latency-related dimensions.The definitions span translation-quality metrics, speech naturalness and consistency, preservation measures, event fidelity, duration consistency, and responsiveness.
  • B.2 Speech Quality and Preservation: Speech quality and preservation are measured using UTMOS, CER/WER, Resemblyzer, WavLM, Emotion2Vec cosine similarity, emotion accuracy, and two event-fidelity F1 metrics.Lower CER/WER indicates better text–speech consistency; higher preservation and fidelity values are better.
  • B.3 Time-related Behavior: The time-related evaluation reports SLC at tolerances 0.2 and 0.4, streaming Start Offset, ATD, Custom ATD, and offline RTF.Higher SLC indicates better duration consistency, whereas lower latency and RTF indicate better responsiveness or efficiency.
  • C Radar Plot Normalization Ranges: Table 9 uses fixed metric ranges for radar normalization so scores remain comparable across systems rather than depending on evaluated-set extremes.Naturally bounded metrics use native ranges, UTMOS uses a MOS-style range, and latency and RTF ranges are clipped for visualization.

D Evaluation Settings

The evaluation uses released or default inference settings and specifies standardized implementations, thresholds, error metrics, native input granularity, and visualization normalization ranges.

  • Inference settings: Most evaluation components use released or default inference settings without additional hyperparameter tuning.Model and implementation choices are specified in Section 3.
  • Evaluation components: The framework uses COMET, BLEURT, Whisper, UTMOS, WavLM, Resemblyzer, emotion2vec, CLAP, and a SimulEval-style interface for evaluation.These components cover translation, transcription, speech quality, speaker similarity, emotion similarity, acoustic-event detection, and latency evaluation.
  • Thresholds and error metrics: SLC is reported at τ = 0.2 and τ = 0.4, with CER for Chinese, Japanese, and Korean targets and WER otherwise.The threshold settings and target-language error metrics are explicitly fixed for evaluation.
  • Latency and visualization: Latency is computed using each system’s native input granularity, while radar plots use fixed normalization ranges only for visualization.Main result tables retain original metric values, and the normalization ranges are listed in Appendix C.
Loading 2605.30792v1…