X-Translator: A Real-Time Multilingual Speaker-Aware Speech-to-Speech Translation System
Abstract
Real-time speech-to-speech translation (S2ST) systems must balance translation quality, latency, speech naturalness, and speaker consistency. Publicly documented S2ST systems have advanced direct, multilingual, streaming, and expressive modeling, while proprietary products and APIs increasingly expose real-time translation capabilities to users. However, practical deployment remains challenging for open and reproducible systems, especially in long-form and multi-speaker conversations where partial ASR hypotheses are unstable, turn boundaries are ambiguous, and target speech must be generated with an appropriate speaker prompt. We present X-Translator, a low-cost modular cascaded S2ST system that combines streaming ASR, machine translation, and prompt-conditioned TTS through a session-level runtime controller. The system uses incremental segment commitment to convert unstable ASR streams into translation-ready units, and an online speaker prompt manager to bind source speech spans to speaker-specific voice prompts for synthesis. We evaluate translation, speech quality, and latency with OpenSTBench, compare against proprietary speech translation APIs as behavioral baselines, measure long-form voice stability, evaluate speaker preservation in multi-speaker conversations, and assess multilingual translation quality. X-Translator provides an open platform for understanding the practical trade-offs of deployment-oriented S2ST. Code and demo are available at https://github.com/zhaoyx239/X-Translator.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
X-Translator presents a practical, modular, and open-source framework for real-time, multi-speaker speech-to-speech translation, offering valuable insights into the system-level engineering challenges of balancing latency, quality, and speaker consistency in deployment-oriented scenarios.
Methodology
The paper proposes X-Translator, a modular, cascaded Speech-to-Speech Translation (S2ST) system designed for real-time, long-form, and multi-speaker scenarios. The core technical contribution lies not in novel model architectures (e.g., new ASR or TTS backbones) but in the system-level runtime controller. Specifically, it introduces an "incremental segment commitment" layer to handle unstable streaming ASR hypotheses and an "online speaker prompt manager" to maintain speaker consistency in multi-speaker conversations. The methodology is pragmatic, leveraging existing state-of-the-art open-source components (Qwen3-ASR, LMT-60-8B, X-Voice) and orchestrating them via explicit runtime policies. While the engineering is sound and addresses real deployment pain points (latency, speaker drift, partial hypothesis instability), the novelty is incremental relative to the broader S2ST literature, which has seen significant advances in direct, end-to-end models (e.g., Translatotron, SeamlessM4T). The paper positions itself as a complementary "deployment-oriented" perspective rather than a fundamental algorithmic breakthrough.
Results & evaluation
The evaluation is comprehensive and well-structured, covering short-form benchmarks (OpenSTBench), long-form single-speaker voice stability, multi-speaker preservation, and multilingual translation. The use of proprietary APIs (Doubao, Qwen LiveTranslate, GPT Realtime Translate) as behavioral baselines is a strong point, providing a realistic comparison against current industry standards. The metrics are appropriate, including translation quality (COMET, BLEU), speech quality (UTMOS), and specific S2ST metrics like speaker similarity (Spk. SIM) and purity measures for multi-speaker settings. The results show that X-Translator achieves competitive speaker preservation and translation quality, though it lags behind proprietary systems in latency and sometimes translation accuracy. The analysis of trade-offs between commitment aggressiveness and quality/latency is insightful. However, the reliance on open-source backbones means the system's performance is bounded by those components, and the "low-cost" aspect comes at the expense of the peak performance seen in proprietary, heavily optimized black-box systems.
Reproducibility
The paper provides a high level of reproducibility. The code and demo are available on GitHub. The modular design allows other researchers to swap out components (ASR, MT, TTS) and test different runtime policies. The detailed description of the segment commitment triggers and speaker prompt management logic is clear. The evaluation protocol is well-documented, including the specific datasets (TEDLIUM-3, WenetSpeech, VoxConverse, FLEURS) and the metrics used. The release of runtime logs for error analysis further enhances reproducibility and interpretability.
Limitations
The paper acknowledges several limitations. First, as a cascaded system, it suffers from error propagation (ASR errors affect MT and TTS) and accumulated latency. Second, the speaker-aware mechanism depends on the quality of online speaker tracking and the availability of clean prompt audio; overlapping speech or rapid turn-taking can degrade performance. Third, the comparison with proprietary systems is behavioral; the internal optimizations of commercial APIs (e.g., joint optimization, larger models, caching) are not visible, making a direct architectural comparison difficult. Finally, the system's performance is limited by the underlying open-source models, which may not match the quality of proprietary counterparts.
Broader impact
X-Translator provides a valuable open-source platform for studying real-time S2ST deployment challenges. By exposing the intermediate decisions (segment commitment, prompt routing), it facilitates research into latency-quality trade-offs and speaker consistency. It democratizes access to real-time S2ST capabilities, allowing researchers to build upon and improve upon existing systems. The focus on multi-speaker and long-form scenarios addresses a significant gap in current S2ST research, which often focuses on short, single-speaker utterances. This work encourages the development of more robust, interpretable, and reproducible speech translation systems. X-Translator presents a practical, modular, and open-source framework for real-time, multi-speaker speech-to-speech translation, offering valuable insights into the system-level engineering challenges of balancing latency, quality, and speaker consistency in deployment-oriented scenarios.
Author of this paper? Send a correction.