One Voice, Many Tongues: Cross-Lingual Voice Cloning for Scientific Speech
Abstract
Preserving a speaker's voice identity while generating speech in a different language remains a fundamental challenge in spoken language technology, particularly in specialized domains such as scientific communication. In this paper, we address this challenge through our system submission to the International Conference on Spoken Language Translation (IWSLT 2026), the Cross-Lingual Voice Cloning shared task. First, we evaluate several state-of-the-art voice cloning models for cross-lingual speech generation of scientific texts in Arabic, Chinese, and French. Then, we build voice cloning systems based on the OmniVoice foundation model. We employ data augmentation via multi-model ensemble distillation from the ACL 60/60 corpus. We investigate the effect of using this synthetic data for fine-tuning, demonstrating consistent improvements in intelligibility (WER and CER) across languages while preserving speaker similarity.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel approach to cross-lingual voice cloning, demonstrating significant advancements in intelligibility and speaker similarity while addressing the challenges of data scarcity in specialized domains. The methodology and results contribute meaningfully to the field of spoken language technology, particularly in the context of scientific communication.
Methodology
The paper employs a robust methodology by leveraging ensemble distillation from multiple state-of-the-art voice cloning models to generate high-fidelity synthetic datasets for fine-tuning. The use of Parameter-Efficient Fine-Tuning (LoRA) is particularly noteworthy, allowing the authors to adapt a large foundation model to specific languages while preserving speaker identity. The approach is well-structured, with clear delineation of the data processing, training configuration, and inference pipeline.
Results & evaluation
The experiments are comprehensive, utilizing a well-defined dataset (ACL 60/60) and a variety of evaluation metrics (WER, CER, and speaker similarity). The results demonstrate consistent improvements in intelligibility and speaker similarity across the three target languages, validating the effectiveness of the proposed methods. The comparative analysis with existing models further strengthens the findings.
Reproducibility
The authors provide a public code repository that includes details on data preparation, training, and evaluation, enhancing the reproducibility of their work. However, the limited scale of the dataset may pose challenges for others attempting to replicate the results at a larger scale.
Limitations
The study is constrained by the size of the distilled training dataset (1,404 samples), which may limit the generalizability of the findings. Additionally, the reliance on automated metrics for evaluation may not fully capture the perceptual quality of synthesized speech, and the paper acknowledges the risks associated with voice cloning technology.
Broader impact
This research has significant implications for enhancing accessibility in scientific communication across different languages, potentially democratizing knowledge dissemination. However, the ethical considerations surrounding voice cloning technologies, such as the potential for misuse, underscore the need for responsible deployment and robust safeguards. The paper presents a novel approach to cross-lingual voice cloning, demonstrating significant advancements in intelligibility and speaker similarity while addressing the challenges of data scarcity in specialized domains. The methodology and results contribute meaningfully to the field of spoken language technology, particularly in the context of scientific communication.
Author of this paper? Send a correction.