SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision
Abstract
Long-form streaming speech-to-speech translation (S2ST) requires incremental, unbounded translation under strict latency constraints. Existing methods typically suffer from sentence-bounded supervision or demand massive paired-S2ST supervision. We introduce a training recipe enabling a speech language model for sentence-level and long-form streaming S2ST using only $\sim$2k hours of paired cross-lingual S2ST data, layered atop auxiliary supervision. Anchored by auxiliary multitask training, our approach remains robust even when the paired-S2ST budget itself is reduced by 90\%. Our core contribution, joint text-code trajectory supervision, schedules target text and acoustic semantic codes as a unified commitment path, eliminating the need for separate, unstable speech-side emission controllers. Furthermore, our two-stream Thinker--Talker factorization significantly outperforms unified-decoder baselines by decoupling linguistic reasoning from dense acoustic prediction to mitigate modality interference. Finally, our system achieves highly competitive quality-latency trade-offs on RealSI and ACL60/60-dev, matching state-of-the-art, closed-source S2ST systems such as LiveInterpret~2.0 on ASR-BLEU.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SimulS2ST-Omni presents a significant technical contribution to streaming speech-to-speech translation by introducing a joint text-code trajectory supervision method and a decoupled two-stream architecture that achieves state-of-the-art performance with substantially reduced data requirements, effectively addressing the critical bottleneck of paired S2ST data scarcity.
Methodology
The paper proposes SimulS2ST-Omni, a data-efficient framework for long-form streaming Speech-to-Speech Translation (S2ST). The core methodological contributions are threefold: (1) A joint text-code trajectory supervision mechanism that schedules target text and acoustic semantic codes simultaneously, eliminating the need for unstable, separate speech-side emission controllers. (2) A two-stream "Thinker-Talker" architecture that decouples linguistic reasoning (Thinker) from dense acoustic prediction (Talker), mitigating modality interference common in unified decoders. (3) A training recipe leveraging auxiliary multitask supervision (ASR, S2TT, MT, TTS) to enable robust performance with only ~2k hours of paired S2ST data, a significant reduction from prior works requiring tens of thousands of hours. The approach extends trajectory-based simultaneous translation to speech output by using cross-lingual word alignments and monotonicity filtering (NIR) to construct stable read/wait/write trajectories.
Results & evaluation
The authors conduct comprehensive experiments on offline S2ST (CVSS-T), sentence-level streaming (RealSI), and long-form streaming (RealSI, ACL60/60-dev). They compare against state-of-the-art public baselines (UniSS, Seamless-M4T) and closed-source systems (LiveInterpret 2.0). Results show that the Thinker-Talker architecture significantly outperforms the matched unified-decoder baseline in translation quality (ASR-BLEU, Text-BLEU) while maintaining competitive or superior speech quality metrics (A.PCP, SIM-O). The system matches or exceeds LiveInterpret 2.0 in several latency tiers. Ablation studies confirm the importance of NIR-based trajectory filtering and the architectural decoupling. The evaluation is rigorous, covering multiple metrics and latency constraints.
Reproducibility
The paper provides detailed descriptions of the data construction process, including the use of public ASR/S2TT corpora to synthesize paired S2ST data, the alignment tools (SimAlign), and the filtering criteria (NIR, ASR error rates). Training hyperparameters, stages, and model initialization details are provided in the appendix. The use of open-source components (Qwen2.5-Omni, DualCodec, Whisper, Paraformer) enhances reproducibility. However, the exact codebase for the trajectory construction and the specific synthesized data pool are not explicitly linked to a public repository, which may hinder exact replication.
Limitations
The experiments are limited to Chinese-English directions due to data scarcity. The paired S2ST data is synthesized rather than naturally recorded simultaneous interpretation, which may not fully capture real-world interpreting phenomena (e.g., disfluencies, interruptions). The speech generation backend (chunk-wise Flow Matching) is not fully streaming-native, potentially limiting inter-chunk coherence and inference efficiency. The work does not address duplex interaction or broader multilingual settings.
Broader impact
This work advances the field of real-time cross-lingual communication by demonstrating that high-quality streaming S2ST is achievable with limited paired data. It addresses ethical concerns regarding voice cloning and impersonation by acknowledging the risks and suggesting safeguards. The data-efficient approach lowers the barrier to entry for developing robust S2ST systems. SimulS2ST-Omni presents a significant technical contribution to streaming speech-to-speech translation by introducing a joint text-code trajectory supervision method and a decoupled two-stream architecture that achieves state-of-the-art performance with substantially reduced data requirements, effectively addressing the critical bottleneck of paired S2ST data scarcity.
Author of this paper? Send a correction.