Learning Timeline-Preserving Speech-to-Speech Translation
Abstract
Speech-to-speech translation (S2ST) increasingly preserves what is said and how it sounds, but not necessarily when speech and silence occur. For dubbing and other temporally grounded applications, matching total duration is not enough: speech and pauses must also occur at corresponding moments. We formulate timeline-preserving S2ST, which aims to preserve the source endpoint and salient internal pauses on a shared absolute timeline. We present TransDubber, an end-to-end autoregressive system that learns this temporal structure directly from aligned target waveforms, without inference-time frame-level timing control or post-hoc temporal correction. To provide this supervision, we construct 642K exactly duration-matched English-Chinese speech pairs by translating and synthesizing target speech on source-defined temporal scaffolds. Training combines a progressive supervised curriculum with verifier-guided reinforcement learning to balance translation quality and temporal alignment. On a 4,353-utterance test set spanning read and spontaneous speech, TransDubber increases the proportion of translations within 2% of the source duration from 25.6% to 61.5% and pause-alignment F1 from 0.118 to 0.550 compared with UniSS, while maintaining competitive translation quality. Our analysis shows that aligned supervision establishes timeline following, while reinforcement learning refines temporal precision and partially offsets the translation-quality cost of strict synchronization. Together, these results demonstrate the effectiveness of our data and training framework for timeline-preserving S2ST. Audio samples, code, datasets, and pretrained models are available at https://zhishengzheng.com/transdubber/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a rigorous framework for timeline-preserving speech-to-speech translation, combining a scalable synthetic data pipeline with component-specific reinforcement learning to achieve state-of-the-art temporal alignment without compromising translation quality.
Methodology
The paper proposes a novel formulation of "timeline-preserving" Speech-to-Speech Translation (S2ST), distinguishing it from prior work that focuses solely on total duration matching or speaker identity. The core methodological contribution is a data construction pipeline that creates 642K exactly duration-matched English-Chinese speech pairs by synthesizing target speech on "source-defined temporal scaffolds." This involves extracting speech/pause regions from source audio, translating segments under local duration budgets, and rendering them back onto the source timeline. The training strategy is a progressive curriculum: (1) pause-aware ASR/translation, (2) natural-pace speech generation, (3) timeline-preserving supervised fine-tuning, and (4) verifier-guided reinforcement learning (GSPO). The RL stage is particularly notable for its component-specific credit assignment, where translation quality rewards are applied to text tokens, while duration and pause alignment rewards are applied to speech-codec tokens, avoiding the dilution of signals that occurs in scalar reward aggregation.
Results & evaluation
The experiments are rigorous, utilizing a 4,353-utterance test set spanning read (CoVoST2) and spontaneous (GigaSpeech) speech. The system significantly outperforms strong baselines (UniSS, Seamless, Cascaded) in temporal metrics: increasing the proportion of translations within 2% of source duration from 25.6% to 61.5% and pause-alignment F1 from 0.118 to 0.550. Crucially, this is achieved without sacrificing translation quality (COMET 0.788 vs 0.784 for UniSS) or speaker similarity. Ablation studies effectively isolate the contributions of pause markers, the natural-pace warm-up stage, and individual reward components, demonstrating that the RL stage specifically refines temporal precision while partially offsetting the quality cost of strict synchronization.
Reproducibility
The paper provides high reproducibility. It details the exact data construction steps, including VAD thresholds, duration budget calculations, and synthesis parameters. The authors commit to releasing code, datasets, and pretrained models. The use of open-source components (Qwen3-ASR, VoxCPM2, Hy-MT2) for the pipeline further aids reproducibility, although the specific "source-canvas rendering" logic is a custom contribution that requires careful implementation.
Limitations
The primary limitation is the reliance on synthetic data for supervision. While the pipeline is scalable, the quality of the synthesized "timeline-preserving" targets depends heavily on the underlying TTS model (VoxCPM2) and the translation model's ability to adhere to strict character budgets. The evaluation is limited to English-Chinese; generalization to other language pairs with different prosodic structures is not tested. Additionally, the "timeline preservation" constraint may not always be desirable in all dubbing contexts, as noted in the related work, though the paper frames this as an application-specific choice.
Broader impact
This work has significant implications for the dubbing industry and real-time speech translation applications where temporal synchronization with visual cues or other audio tracks is critical. By demonstrating that autoregressive models can learn complex temporal structures directly from aligned waveforms without explicit frame-level control at inference, it opens the door to more natural and synchronized multilingual audio generation. The component-specific RL approach could be adapted to other multimodal generation tasks requiring precise alignment between different modalities. The paper introduces a rigorous framework for timeline-preserving speech-to-speech translation, combining a scalable synthetic data pipeline with component-specific reinforcement learning to achieve state-of-the-art temporal alignment without compromising translation quality.
Author of this paper? Send a correction.