UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Models
Abstract
Spoken dialogue models currently lack the ability for fine-grained speech style control, a critical capability for human-like interaction that is often overlooked in favor of purely functional capabilities like reasoning and question answering. To address this limitation, we introduce UltraVoice, the first large-scale speech dialogue dataset engineered for multiple fine-grained speech style control. Encompassing over 830 hours of speech dialogues, UltraVoice provides instructions across six key speech stylistic dimensions: emotion, speed, volume, accent, language, and composite styles. Fine-tuning leading models such as SLAM-Omni and VocalNet on UltraVoice significantly enhances their fine-grained speech stylistic controllability without degrading core conversational abilities. Specifically, our fine-tuned models achieve improvements of 29.12-42.33% in Mean Opinion Score (MOS) and 14.61-40.09 percentage points in Instruction Following Rate (IFR) on multi-dimensional control tasks designed in the UltraVoice. Moreover, on the URO-Bench benchmark, our fine-tuned models demonstrate substantial gains in core understanding, reasoning, and conversational abilities, with average improvements of +10.84% on the Basic setting and +7.87% on the Pro setting. Furthermore, the dataset's utility extends to training controllable Text-to-Speech (TTS) models, underscoring its high quality and broad applicability for expressive speech synthesis. The complete dataset and model checkpoints are available at: https://github.com/bigai-nlco/UltraVoice.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of the UltraVoice dataset, which enables fine-grained style control in spoken dialogue models, thereby enhancing the expressiveness and human-likeness of speech interactions. This work is significant as it addresses a critical gap in the current capabilities of spoken dialogue systems and provides a valuable resource for future research in the field.
Methodology
The methodology presented in the paper is robust, focusing on the creation of a large-scale synthetic dataset, UltraVoice, which is designed to facilitate fine-grained style control in spoken dialogue systems. The use of GPT-4o for text generation and various TTS engines for audio synthesis is innovative, allowing for the generation of diverse speech styles without the ethical concerns associated with using real human voices. The paper clearly outlines the dimensions of style control and the models used for fine-tuning, which enhances reproducibility.
Results & evaluation
The experiments conducted demonstrate significant improvements in both stylistic controllability and core conversational abilities of the models fine-tuned on the UltraVoice dataset. The reported improvements in Mean Opinion Score (MOS) and Instruction Following Rate (IFR) are substantial and indicate that the proposed dataset effectively enhances model performance. The use of benchmarks like URO-Bench adds credibility to the evaluation, although more details on the experimental setup would strengthen the findings.
Reproducibility
The authors provide a comprehensive reproducibility statement, detailing the data generation pipeline, model configurations, and evaluation metrics. This transparency is commendable and facilitates further research. However, the lack of specific institution information may hinder some aspects of reproducibility in terms of institutional resources.
Limitations
While the paper addresses ethical concerns related to the misuse of synthetic speech, it does not delve deeply into the limitations of the dataset itself, such as potential biases in the generated speech styles or the generalizability of the models trained on synthetic data. Additionally, the reliance on synthetic data may limit the realism of the conversational interactions.
Broader impact
The UltraVoice dataset has the potential to significantly advance the field of controllable speech synthesis and dialogue systems, enabling more human-like interactions in various applications, including virtual assistants and entertainment. However, the potential for misuse, such as creating deepfakes or manipulating emotions through stylized speech, raises important ethical considerations that the authors have acknowledged. The main contribution of this paper is the introduction of the UltraVoice dataset, which enables fine-grained style control in spoken dialogue models, thereby enhancing the expressiveness and human-likeness of speech interactions. This work is significant as it addresses a critical gap in the current capabilities of spoken dialogue systems and provides a valuable resource for future research in the field.
Author of this paper? Send a correction.