DuraS2ST: Chain-of-Thought and Reinforcement Learning for Duration-Aligned Speech-to-Speech Translation
Abstract
Speech-to-speech translation (S2ST) in time-sensitive applications such as video dubbing requires not only semantic fidelity and speaker preservation, but also strict duration consistency to avoid audio-visual misalignment. However, existing S2ST systems largely generate target speech without explicit temporal planning, making duration control an unresolved challenge. We introduce DuraS2ST, a duration-aligned reasoning framework that enables a single speech language model to first generate an explicit chain-of-thought (CoT) for planning target wording and phonetic length, and then synthesize the corresponding speech tokens. To support this paradigm, we construct DuraSet-440K, a high-quality duration-aligned CoT corpus for supervised initialization. We further optimize the model with multi-modal multi-dimensional reinforcement learning, using a Duration Margin Reward to balance translation quality and duration consistency, and Modality-Aware Reward Attribution to assign rewards to appropriate token spans. Experiments on CVSS-T show that DuraS2ST achieves a strong balance between translation quality and duration consistency, outperforming competitive open-source and commercial baselines. Project page: https://github.com/Mia11939/DuraS2ST.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces DuraS2ST, a novel framework that integrates Chain-of-Thought reasoning and reinforcement learning to achieve duration-aligned speech-to-speech translation. By constructing a dedicated corpus (DuraSet-440K) and designing modality-aware reward mechanisms (DMR and MARA), the authors effectively address the challenge of temporal consistency in S2ST, demonstrating superior performance over strong commercial and open-source baselines on the CVSS-T benchmark.
Methodology
The paper proposes DuraS2ST, a framework that reformulates duration-aligned speech-to-speech translation (S2ST) as a reasoning problem rather than an acoustic post-processing task. The core methodological contribution is the integration of Chain-of-Thought (CoT) reasoning into the speech generation pipeline. The model first generates an explicit textual rationale planning the target wording and phonetic length, followed by the synthesis of interleaved text-acoustic tokens. This is supported by the construction of DuraSet-440K, a large-scale corpus specifically designed for this purpose. The training paradigm is two-phase: Supervised Fine-Tuning (SFT) on the new corpus, followed by Group Relative Policy Optimization (GRPO). The RL phase introduces two specific technical innovations: the Duration Margin Reward (DMR), which acts as a soft constraint to balance translation quality with duration consistency, and Modality-Aware Reward Attribution (MARA), which prevents cross-modality reward contamination by assigning duration rewards only to acoustic tokens and quality rewards to both text and acoustic spans. This approach is technically sound and addresses a specific gap in current S2ST systems where duration control is often handled via black-box speed tokens or post-hoc time-stretching.
Results & evaluation
The experiments are conducted on the CVSS-T benchmark, which is disjoint from the training data, providing a valid out-of-domain evaluation. The paper compares DuraS2ST against strong baselines, including commercial models (GPT-4o, Qwen2.5-Omni, Kimi-Audio) and open-source SLMs (Step-Audio-2-mini). The results demonstrate that DuraS2ST achieves a superior balance between translation quality (BLEU, COMET) and duration consistency (SLC-0.2, SLC-0.4, MADE, MRDE). Notably, the method shows substantial gains in a zero-shot RL setting, highlighting the effectiveness of the reward design. The inclusion of both open-source and commercial baselines strengthens the experimental validation.
Reproducibility
The paper provides a project page with a GitHub repository link. It details the training setup, including hyperparameters for SFT and GRPO, and describes the construction pipeline for DuraSet-440K in detail. The use of standard metrics (BLEU, COMET, SECS, SLC) and a public benchmark (CVSS-T) enhances reproducibility. However, the specific implementation details of the "duration-controllable TTS model" used for data synthesis are not fully specified in the main text, though likely detailed in the appendix or code.
Limitations
The method relies on a specific base model (Step-Audio-2-mini-Think), and its generalizability to other SLM architectures is not extensively tested. The construction of DuraSet-440K is resource-intensive, requiring LLM-based translation candidate generation and duration-controllable TTS synthesis. The evaluation is limited to English-Chinese pairs; performance on other language pairs with different phonetic structures remains unknown. Additionally, the "zero-shot RL" claim needs careful interpretation as it likely refers to the RL phase without additional SFT on the target domain, but the model is still SFT'd on DuraSet-440K.
Broader impact
This work has significant implications for video dubbing, simultaneous interpretation, and real-time communication applications where audio-visual synchronization is critical. By treating duration alignment as a reasoning task, it offers a more interpretable and controllable approach compared to traditional acoustic constraints. The release of DuraSet-440K and the code will likely accelerate research in duration-controlled speech generation and multimodal reasoning. The paper introduces DuraS2ST, a novel framework that integrates Chain-of-Thought reasoning and reinforcement learning to achieve duration-aligned speech-to-speech translation. By constructing a dedicated corpus (DuraSet-440K) and designing modality-aware reward mechanisms (DMR and MARA), the authors effectively address the challenge of temporal consistency in S2ST, demonstrating superior performance over strong commercial and open-source baselines on the CVSS-T benchmark.
Author of this paper? Send a correction.