MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions
Abstract
Voice design from natural language aims to generate speaker timbres directly from free-form textual descriptions, allowing users to create voices tailored to specific roles, personalities, and emotions. Such controllable voice creation benefits a wide range of downstream applications-including storytelling, game dubbing, role-play agents, and conversational assistants, making it a significant task for modern Text-to-Speech models. However, existing models are largely trained on carefully recorded studio data, which produces speech that is clean and well-articulated, yet lacks the lived-in qualities of real human voices. To address these limitations, we present MOSS-VoiceGenerator, an open-source instruction-driven voice generation model that creates new timbres directly from natural language prompts. Motivated by the hypothesis that exposure to real-world acoustic variation produces more perceptually natural voices, we train on large-scale expressive speech data sourced from cinematic content. Subjective preference studies demonstrate its superiority in overall performance, instruction-following, and naturalness compared to other voice design models.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
MOSS-VoiceGenerator presents an innovative approach to generating realistic voices from natural language descriptions, addressing key limitations in existing TTS systems. The combination of a diverse training dataset and advanced model architecture positions this work as a significant contribution to the field of audio synthesis and voice generation.
Methodology
The methodology presented in MOSS-VoiceGenerator is robust and innovative, leveraging a large-scale dataset derived from cinematic content to train a model that generates realistic voices from natural language descriptions without the need for reference audio. The authors employ a comprehensive data collection and annotation process, ensuring a diverse and expressive dataset. The model architecture integrates autoregressive techniques with a discrete framework, which simplifies deployment and enhances instruction-following capabilities. However, the reliance on a specific dataset type (cinematic) may limit generalizability to other domains.
Results & evaluation
The experimental evaluation is thorough, utilizing both subjective and objective metrics to assess the model's performance. The inclusion of a public benchmark (InstructTTSEval) for objective evaluation adds credibility to the results. The subjective preference studies provide valuable insights into user experience and model performance across different dimensions. The results indicate that MOSS-VoiceGenerator outperforms several existing models, showcasing its effectiveness in generating expressive and natural-sounding speech.
Reproducibility
The paper outlines the training strategy and data processing pipeline in detail, which aids in reproducibility. However, the lack of a publicly accessible demo or project URL limits the ability for other researchers to replicate the work easily. Open-sourcing the model and data pipeline would significantly enhance reproducibility and community engagement.
Limitations
The authors acknowledge several limitations, including the focus on Chinese and English, which restricts language diversity. The English dataset is smaller, potentially affecting performance in English voice generation. Additionally, the denoising process may introduce artifacts, and the model's output can occasionally lack stability. These limitations suggest areas for future improvement and expansion.
Broader impact
MOSS-VoiceGenerator has significant potential applications in various domains such as audiobook narration, game dubbing, and conversational agents, where realistic and expressive voice generation is crucial. The open-source nature of the project could foster further research and development in controllable TTS systems, contributing to advancements in human-computer interaction and accessibility. MOSS-VoiceGenerator presents an innovative approach to generating realistic voices from natural language descriptions, addressing key limitations in existing TTS systems. The combination of a diverse training dataset and advanced model architecture positions this work as a significant contribution to the field of audio synthesis and voice generation.
Author of this paper? Send a correction.