NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion
Abstract
Everyday speech conveys far more than words, it reflects who we are, how we feel, and the circumstances surrounding our interactions. Yet, most existing speech datasets are acted, limited in scale, and fail to capture the expressive richness of real-life communication. With the rise of large neural networks, several large-scale speech corpora have emerged and been widely adopted across various speech processing tasks. However, the field of voice conversion (VC) still lacks large-scale, expressive, and real-life speech resources suitable for modeling natural prosody and emotion. To fill this gap, we release NaturalVoices (NV), the first large-scale spontaneous podcast dataset specifically designed for emotion-aware voice conversion. It comprises 5,049 hours of spontaneous podcast recordings with automatic annotations for emotion (categorical and attribute-based), speech quality, transcripts, speaker identity, and sound events. The dataset captures expressive emotional variation across thousands of speakers, diverse topics, and natural speaking styles. We also provide an open-source pipeline with modular annotation tools and flexible filtering, enabling researchers to construct customized subsets for a wide range of VC tasks. Experiments demonstrate that NaturalVoices supports the development of robust and generalizable VC models capable of producing natural, expressive speech, while revealing limitations of current architectures when applied to large-scale spontaneous data. These results suggest that NaturalVoices is both a valuable resource and a challenging benchmark for advancing the field of voice conversion. Dataset is available at: https://huggingface.co/JHU-SmileLab
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of NaturalVoices, a large-scale, spontaneous podcast dataset designed for emotion-aware voice conversion, which addresses critical gaps in existing datasets and provides a valuable resource for advancing research in voice conversion and emotional speech synthesis. This work significantly enhances the field by offering a comprehensive dataset that captures the richness of real-world speech, along with a robust methodology for its utilization in voice conversion tasks.
Methodology
The methodology presented in the paper is robust, focusing on the development of the NaturalVoices dataset, which is a significant advancement in the field of voice conversion. The authors provide a comprehensive automated data-sourcing pipeline that includes multi-level annotations for emotion, speaker identity, and speech quality. This systematic approach allows for flexible filtering and customization of the dataset, making it suitable for various voice conversion tasks. The use of podcasts as a source of spontaneous speech is particularly innovative, as it captures the natural emotional dynamics that are often absent in traditional datasets.
Results & evaluation
The experimental evaluation is thorough, demonstrating the dataset's utility through various voice conversion models. The authors conduct extensive experiments to assess the performance of state-of-the-art models trained on NaturalVoices, revealing both the strengths and limitations of current architectures. The results indicate that NaturalVoices not only supports high-quality voice conversion but also serves as a challenging benchmark, highlighting the need for more advanced models capable of leveraging the dataset's scale and variability.
Reproducibility
The paper provides adequate details regarding the dataset construction and the automated processing pipeline, which enhances reproducibility. The authors have made the dataset publicly available along with the necessary tools for annotation and filtering, which facilitates further research. However, specific implementation details regarding the models used in experiments could be expanded to improve reproducibility.
Limitations
One limitation identified in the paper is the variability in audio quality due to the nature of podcast recordings, which may affect the performance of voice conversion models. Additionally, while the dataset is extensive, it may still lack certain emotional nuances that could be captured through more diverse sources or additional annotations. The authors also acknowledge that current architectures struggle to fully exploit the dataset's complexity, indicating a gap in model capabilities.
Broader impact
The potential applications of NaturalVoices are significant, ranging from conversational speech synthesis to affective computing and deepfake detection. By providing a rich, expressive dataset, this work paves the way for advancements in voice conversion technologies that can better capture the nuances of human communication. The implications extend to various fields, including entertainment, accessibility, and human-computer interaction. The main contribution of this paper is the introduction of NaturalVoices, a large-scale, spontaneous podcast dataset designed for emotion-aware voice conversion, which addresses critical gaps in existing datasets and provides a valuable resource for advancing research in voice conversion and emotional speech synthesis. This work significantly enhances the field by offering a comprehensive dataset that captures the richness of real-world speech, along with a robust methodology for its utilization in voice conversion tasks.
Author of this paper? Send a correction.