Source-linked AI summary
X-Translator: A Real-Time Multilingual Speaker-Aware Speech-to-Speech Translation System
Yuxiang Zhao, Yichi Zhang, Yanjie An, Yanqiao Zhu, Zhanxun Liu, Yushen Chen, Qixi Zheng, Haina Zhu, Yunchong Xiao, Keqi Deng, Shuai Fan, Kai Yu, Xie Chen
TL;DR
Real-time S2ST needs to balance translation quality, latency, speech naturalness, and speaker consistency, particularly in long-form multi-speaker conversations. X-Translator uses a modular cascaded runtime with incremental segment commitment and speaker-specific prompt routing. It provides strong speaker preservation and low target-speech error rates in evaluated short-form settings, but does not consistently surpass proprietary systems in latency or translation quality.
Problem
Real-time S2ST must balance translation quality, latency, natural target speech, and speaker consistency, while long-form multi-speaker deployment remains challenging.
Method
X-Translator combines streaming ASR, MT, and prompt-conditioned TTS through a session-level controller that commits stable segments and routes speaker-specific prompts.
Results
X-Translator provides strong speaker preservation and low target-speech error rates in evaluated short-form settings, but its latency and translation quality do not consistently surpass proprietary systems.
Takeaways & Limitations
X-Translator provides an open and controllable platform for studying how segment commitment, prompt routing, and module-level timing shape deployment-oriented S2ST.
Takeaways & Limitations
Cascaded errors can propagate across ASR, MT, and TTS, while separate training prevents joint optimization of translation quality, speaker similarity, prosody, and latency.
Abstract
from arXiv · showhide
Real-time speech-to-speech translation (S2ST) systems must balance translation quality, latency, speech naturalness, and speaker consistency. Publicly documented S2ST systems have advanced direct, multilingual, streaming, and expressive modeling, while proprietary products and APIs increasingly expose real-time translation capabilities to users. However, practical deployment remains challenging for open and reproducible systems, especially in long-form and multi-speaker conversations where partial ASR hypotheses are unstable, turn boundaries are ambiguous, and target speech must be generated with an appropriate speaker prompt. We present X-Translator, a low-cost modular cascaded S2ST system that combines streaming ASR, machine translation, and prompt-conditioned TTS through a session-level runtime controller. The system uses incremental segment commitment to convert unstable ASR streams into translation-ready units, and an online speaker prompt manager to bind source speech spans to speaker-specific voice prompts for synthesis. We evaluate translation, speech quality, and latency with OpenSTBench, compare against proprietary speech translation APIs as behavioral baselines, measure long-form voice stability, evaluate speaker preservation in multi-speaker conversations, and assess multilingual translation quality. X-Translator provides an open platform for understanding the practical trade-offs of deployment-oriented S2ST. Code and demo are available at https://github.com/zhaoyx239/X-Translator.
1 Introduction
Real-time S2ST must jointly address translation quality, latency, speech naturalness, and speaker consistency, but open deployment remains difficult in long-form and multi-speaker settings. X-Translator responds with an open modular cascade, runtime controls for stable segment commitment and speaker prompting, and deployment-oriented evaluation.
- Real-time S2ST must balance translation quality, low latency, natural target speech, and speaker consistency in multi-speaker audio.
- Proprietary systems demonstrate practical potential, but their closed architectures and data limit reproducibility, adaptation, and component-level diagnosis.
- X-Translator composes replaceable ASR, MT, and prompt-conditioned TTS modules through a session-level runtime controller.
- The evaluation combines OpenSTBench with long-form, multi-speaker, and multilingual experiments to measure runtime stability, generalization, and speaker-aware voice behavior.
- Incremental segment commitment converts unstable streaming ASR hypotheses into translation-ready units without forwarding display-only partial hypotheses downstream.
- An online speaker prompt manager maintains speaker-specific state and routes committed segments to appropriate synthesis prompts.
2 Related Work
Related work spans cascaded and direct S2ST, multilingual and streaming systems, proprietary behavioral baselines, speaker-aware synthesis, and broader evaluation protocols. X-Translator focuses on inspectable runtime decisions and extends short-form comparison to long-form and multi-speaker behavior.
- Cascaded and direct S2ST: Cascaded S2ST combines ASR, MT, and TTS for modularity and inspection, while direct systems map source speech to target speech more directly.
- Multilingual and streaming S2ST: Large-scale systems such as SeamlessM4T broaden S2ST toward multilingual, expressive, and streaming speech translation.
- Proprietary systems: Commercial systems are treated as proprietary behavioral baselines evaluated through observable text or speech outputs, latency, and stability.
- Streaming translation: Streaming S2ST must decide when to emit output despite changing ASR hypotheses, creating a latency-quality tension addressed here through runtime-level stable segment commitment.
- Voice preservation and speaker-aware synthesis: Long-form multi-speaker translation requires assigning each source segment to a speaker-specific synthesis prompt, implemented here with speaker prompt buffers and routing.
- Evaluation: OpenSTBench supports short-form comparison, while long-form and multi-speaker tests expose repeated, dropped, delayed, unstable, or incorrectly routed outputs.
3 X-Translator
X-Translator is a modular cascaded real-time S2ST system that exposes runtime decisions for ASR commitment, speaker routing, prompt selection, and latency measurement. Its controller turns unstable streaming hypotheses into committed units, binds them to speaker-specific prompts, and sends them through translation and synthesis.
- 3.1 System Overview: X-Translator streams audio through a session controller that keeps partial ASR text display-only while sending only committed segments to MT and TTS.The server tracks hypotheses, committed segments, translation and TTS requests, speaker prompts, and playback metadata.
- 3.2 Segment Commitment: Commitment triggers cut stable prefixes at punctuation, silence or no-update timeouts, and speaker boundaries, making each committed event a downstream translation unit.Each event contains a time span, speaker identifier, and source text.
- 3.3 Speaker-Aware Prompt Management: The speaker prompt manager assigns each committed segment to the speaker with the greatest temporal overlap between the ASR span and speaker intervals.This overlap rule handles segments whose boundaries do not exactly match speaker changes.
- 3.3 Speaker-Aware Prompt Management: Each speaker maintains fixed and rolling six-second prompts, and the configured synthesis mode selects one before translated text and prompt audio are sent to TTS.When a complete prompt is unavailable, available speaker audio or a default voice can be used.
- 3.4 Cascaded Processing: Serial ASR, MT, and TTS processing treats committed segments as atomic units while preserving ordering, failure isolation, and browser playback queue management.Only committed segments trigger target speech generation, avoiding repeated playback from ASR revisions.
- 3.5 Runtime Modularity: X-Translator logs ASR commitment, MT, TTS, and playback times to decompose end-to-end delay and support behavioral comparisons under a shared streaming policy.The modular system is replaceable and interpretable, but cascades can accumulate ASR errors, segmentation mismatches, and network latency.
4 Experiment Setup
X-Translator is evaluated across short-form, long-form, multi-speaker, and multilingual settings using fixed backends, proprietary behavioral baselines, and metrics spanning quality, latency, voice stability, and speaker consistency.
- Evaluation tasks: The evaluation covers short-form OpenSTBench, long-form single-speaker streaming, long-form multi-speaker conversations, and multilingual speech translation.Long-form tests use at least ten-minute recordings; multilingual evaluation includes 16 input and 19 output languages supported by the evaluated backend stack.
- Systems and baselines: X-Translator uses a fixed Qwen3-ASR, LMT-60-8B, and X-Voice backend stack, while proprietary comparisons focus on observable outputs and timing behavior.Baseline short-form evaluations are run twice and averaged because some APIs showed unstable availability or variable response behavior.
- Short-form metrics: Short-form S2ST quality is measured from target-speech ASR transcripts using BLEU, chrF++, COMET, and BLEURT, alongside CER/WER, latency, speaker similarity, and emotion similarity.S2TT metrics are computed separately from directly returned translation text before synthesis and target-speech ASR.
- Long-form voice stability: Long-form single-speaker evaluation compares source and target voice embeddings over 10-second windows with 5-second hops using SSLG, TTLG, and GSS measures.Timelines are normalized by cumulative effective speech duration after removing non-speech regions with VAD; long-gap similarity uses γ = 0.5.
- Multi-speaker evaluation: Multi-speaker evaluation aligns source and target segments semantically, constrains matches by relative temporal position, and measures voice similarity plus speaker purity.Matched segment voice similarity uses Resemblyzer embeddings and a source-duration-weighted average; correspondence matrices support speaker consistency analysis.
5 Results
X-Translator shows strong speaker preservation and voice stability, while trading off translation quality and latency against proprietary systems. Its multilingual and multi-speaker results indicate broad coverage with direction- and speaker-dependent variation.
- Short-Form Speech Translation: X-Translator achieves the highest Spk. SIM in both directions: 0.87 for English-to-Chinese and 0.88 for Chinese-to-English.
- Short-Form Speech Translation: X-Translator is the slowest system in both directions, with start offsets of 2,584 and 4,056 ms.GPT Realtime Translate records 814 and 800 ms in the corresponding directions.
- Short-Form Speech Translation: X-Translator’s COMET scores are 0.78 and 0.74, below the strongest proprietary baselines in both translation directions.For English-to-Chinese, Qwen3-LiveTranslate reaches 0.81 and Doubao AST 2.0 reaches 0.79; for Chinese-to-English, Doubao reaches 0.77.
- Long-Form Voice Stability: X-Translator’s target voice remains internally stable, with TTLG scores of 0.91 and 0.90, while GSS reaches 0.83 and 0.79.These values indicate long-range target consistency alongside substantial source-speaker similarity.
- Multi-Speaker Speech Translation: In multi-speaker sessions, X-Translator reaches Spk. SIM of 0.77 in both directions and remains within 0.02–0.07 of Doubao’s purity scores.Its Target Purity is 0.85 and 0.84, while Source Purity is 0.93 and 0.88.
- Multilingual Speech Translation: Across 70 FLEURS directions, X-Translator obtains a macroaveraged COMET of 0.69, with group averages ranging from 0.68 to 0.70.The macro-average reflects broad backend coverage rather than uniformly strong performance across every language pair.
6 Limitations and Future Work
X-Translator inherits cascade, runtime-policy, speaker-tracking, and black-box-comparison limitations. Future work targets stronger commitment, speaker routing, prompt selection, and broader evaluation conditions.
- Limitations: ASR errors can propagate through MT and TTS, while early segment commitment may discard context that later modules cannot recover.
- Limitations: Because the modules are not trained end to end, X-Translator cannot jointly optimize translation quality, speaker similarity, prosody, and latency.Explicit commitment and prompt-routing policies improve controllability but introduce policy-sensitive failure modes.
- Limitations: Overlapping speech, rapid turn-taking, noise, similar voices, and weak or mismatched prompts can make speaker assignment and synthesis unreliable.Early prompt errors may affect many later synthesized segments in long-form conversations.
- Limitations: Comparisons with proprietary systems are behavioral rather than mechanistic because their models, data, caching, endpointing, and safety filters are hidden.
- Future Work: Future work will combine lexical, acoustic, prosodic, and uncertainty cues for commitment and improve speaker routing, prompt quality, prompt selection, and evaluation breadth.Planned evaluations include more language pairs, noisier settings, longer conversations, and additional baselines.
7 Conclusion
X-Translator is an open, controllable cascaded S2ST system for real-time, long-form, and multi-speaker translation. It exposes deployment decisions for studying trade-offs that short-form evaluation alone does not capture.
- Conclusion: X-Translator combines streaming ASR, MT, and prompt-conditioned TTS through a session-level controller that commits stable segments and routes speaker-specific prompts.
- Conclusion: The system targets real-time, long-form, and multi-speaker settings whose deployment constraints are not fully captured by short-form sentence-level evaluation.The long-form protocol continuously feeds ten recordings per dataset to simulate streaming input.
- Conclusion: X-Translator provides strong speaker preservation and low target-speech error rates, but its latency and translation quality do not consistently surpass proprietary systems.
- Conclusion: Exposed segment commitment, prompt routing, and module-level timing support practical study of how runtime decisions shape deployment-oriented S2ST.
B Multi-Speaker Dataset Details
The multi-speaker evaluation documents its source audio and treats speaker prompts as transient session-conditioning data rather than released identity labels.
- Dataset Details: Table 8 documents the multi-speaker evaluation sources and their actual audio durations.
- Dataset Details: Speaker prompts are transient excerpts from each evaluated source recording and are used only to condition synthesis within that session.
- Dataset Details: Inferred speaker identities are not treated as labels, and source-derived voice prompts are not released.
C Implementation and Runtime Configuration
The released evaluation configuration documents the backend selection and connects three generative modules through localhost endpoints. It uses one multilingual language-pair worker by default, without request batching or an application-level result cache.
- Table 9 documents the backend selection encoded by the released evaluation configuration.
- The configuration connects three generative modules through localhost endpoints and runs one multilingual language-pair worker by default.
- Request batching and an application-level result cache are not enabled.
D Multilingual Evaluation Details
The multilingual evaluation transcribes generated target speech with explicitly configured target-language ASR before computing COMET. It covers broad input and output language sets, reports macro-averages over successful samples with separate coverage, and excludes same-language pairs.
- Evaluation metric: Generated speech is transcribed with Whisper-medium using an explicit target-language code before COMET computation.
- Language coverage: The final language coverage includes 18 input languages and 21 output languages, counting Chinese and English.
- Evaluation metric: COMET is reported over successful samples, while coverage is retained separately and failed samples receive zero in a conservative all-sample score.
- Aggregation: The direction-group macro-average gives every direction equal weight, and same-language pairs are excluded.
E Proprietary Baseline Configuration
The proprietary systems are treated as time-dependent behavioral baselines identified by requested API model and evaluation setting. Short-form and long-form collections used different Qwen service versions, and successful outputs required playable speech plus semantically complete translation.
- Baseline framing: Proprietary systems are documented as time-dependent behavioral baselines rather than inaccessible internal checkpoints.
- Service versions: Qwen3-LiveTranslate was used for short-form OpenSTBench collection, whereas Qwen3.5-LiveTranslate was used for later long-form experiments.
- Scope boundary: Cross-table short-form and long-form comparisons should not be interpreted as comparisons under an identical Qwen baseline.
- Evaluation procedure: Successful outputs required non-empty playable speech and a semantically complete translation under OpenSTBench checks.
F OpenSTBench API Success Rates
Short-form OpenSTBench success rates measure calls that return playable speech and semantically complete translations. Lower success rates for Qwen3-LiveTranslate and Doubao AST 2.0 in some settings mainly reflect semantic-completeness filtering of short audio rather than transport failure.
- Success-rate reporting: Table 12 reports successful-call rates by dataset and translation direction for the short-form OpenSTBench evaluation.
- Interpretation: Lower success rates for Qwen3-LiveTranslate and Doubao AST 2.0 in some settings are mainly caused by short-audio semantic-completeness filtering.
- Success definition: For proprietary systems, an unsuccessful call includes a short input lacking a semantically complete translation, not only a transport failure.