Timbre-Aware LLM-based Direct Speech-to-Speech Translation Extendable to Multiple Language Pairs
Abstract
Direct Speech-to-Speech Translation (S2ST) has gained increasing attention for its ability to translate speech from one language to another, while reducing error propagation and latency inherent in traditional cascaded pipelines. However, existing direct S2ST systems continue to face notable challenges, including instability in semantic-acoustic alignment when parallel speech data is scarce, difficulty in preserving speaker identity, and limited multilingual scalability. In this work, we introduce DS2ST-LM, a scalable, single-stage direct S2ST framework leveraging a multilingual Large Language Model (LLM). The architecture integrates a Whisper speech encoder, a learnable projection module, a Qwen2-0.5B LLM, and a timbre-controlled vocoder. We construct GigaS2S-1000, a 1000-hour bilingual corpus by extending the GigaST dataset with high-fidelity synthetic target speech, and show that this synthetic data alleviates data scarcity to some extent. We investigate two semantic token generation strategies: speech-derived S3 tokens and text-derived tokens generated by a pre-trained LLM, and analyze their impact on training stability and semantic consistency. We further evaluate three projection architectures (Linear, Conv1D-Linear, and Q-Former) and observe that while higher-capacity projectors converge faster, the simple Linear projector achieves higher performance. Extensive experiments demonstrate that DS2ST-LM outperforms traditional cascaded and ST (Qwen-Audio) + TTS baselines across both lexical (BLEU, METEOR) and semantic (BLEURT, COMET) metrics, while extending to multiple language pairs, including French, Spanish, German, Hindi, Bengali, and Urdu. Furthermore, we incorporate timbre-aware speech synthesis to preserve speaker information, enabling DS2ST-LM to surpass prior direct S2ST systems in both speaker similarity and perceptual naturalness.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this work is the development of DS2ST-LM, a scalable direct speech-to-speech translation framework that effectively addresses key challenges in the field while demonstrating superior performance across multiple language pairs. The comprehensive methodology, rigorous experimental evaluation, and commitment to reproducibility position this research as a significant advancement in the S2ST domain, with potential applications that extend beyond traditional language barriers.
Methodology
The proposed methodology introduces a novel single-stage direct speech-to-speech translation (S2ST) framework, DS2ST-LM, which integrates a Whisper speech encoder, a multilingual large language model (LLM), and a timbre-controlled vocoder. The architecture is innovative in its use of semantic token generation strategies, comparing speech-derived and text-derived tokens, and evaluating various projection architectures to optimize performance. The incorporation of a large-scale synthetic dataset, GigaS2S-1000, enhances the model's training capabilities, addressing the scarcity of parallel speech data. The approach is well-structured, systematically addressing key challenges in S2ST, such as semantic-acoustic alignment and speaker identity preservation.
Results & evaluation
The experimental evaluation is comprehensive, utilizing multiple datasets and baseline comparisons to assess the performance of DS2ST-LM. The results demonstrate significant improvements over traditional cascaded systems and ST + TTS baselines across various lexical and semantic metrics. The paper provides detailed performance metrics, including BLEU, METEOR, BLEURT, and COMET, which effectively illustrate the model's strengths in translation quality and speaker similarity. The extensive evaluation across multiple language pairs further validates the model's robustness and scalability.
Reproducibility
The authors emphasize reproducibility by releasing all training recipes, evaluation pipelines, and model checkpoints. This commitment to transparency is commendable and facilitates further research in the field. However, the paper could benefit from more detailed implementation instructions or a dedicated repository for code and models to enhance accessibility for other researchers.
Limitations
While the paper presents a strong framework, limitations include the reliance on synthetic data for training, which may not fully capture the nuances of natural speech. Additionally, the performance variations across languages suggest that the model's effectiveness may be contingent on the linguistic diversity and representation in the training data. The study also does not address potential biases in the synthetic data generation process, which could impact the model's generalization to real-world applications.
Broader impact
The proposed DS2ST-LM framework has significant implications for real-time multilingual communication, potentially enhancing accessibility and understanding in diverse linguistic contexts. The ability to preserve speaker identity and produce high-quality speech synthesis could facilitate applications in areas such as international diplomacy, global business, and education. Furthermore, the research contributes to the growing body of work on integrating LLMs in speech processing, paving the way for future advancements in the field. The main contribution of this work is the development of DS2ST-LM, a scalable direct speech-to-speech translation framework that effectively addresses key challenges in the field while demonstrating superior performance across multiple language pairs. The comprehensive methodology, rigorous experimental evaluation, and commitment to reproducibility position this research as a significant advancement in the S2ST domain, with potential applications that extend beyond traditional language barriers.
Author of this paper? Send a correction.