Toward Natural Emotional Text-To-Speech System with Fine-Grained Non-Verbal Expression Control
Abstract
While current emotional Text-to-Speech (TTS) models have successfully controlled verbal prosody, they often ignore non-verbal vocalizations (NVs), which are essential for authentic human emotion. Although some non-verbal datasets have recently emerged, they often lack high-quality, fine-grained annotations, which restricts a model's ability to precisely control NV generation. To address this limitation, we propose a novel approach for fine-grained non-verbal expression synthesis. We curate and reprocess female NV utterances from the EARS corpus, develop a new annotation scheme using tags to encode NV types, frequencies, and durations, and build an emotional TTS benchmark to demonstrate its effectiveness. Our evaluation shows that while our NV approach leads to minor trade-offs in perceived naturalness, it significantly improves expressiveness (eMOS 4.20) and emotional recognition accuracy (78.8%). Emotion-specific analysis further reveals that NV cues are highly effective for high-arousal emotions like happy (82.5%) and fear (82.7%), and almost perfectly convey sadness (98.3%).
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of a fine-grained non-verbal expression dataset and a corresponding TTS system that significantly enhances emotional expressiveness in synthesized speech. This work represents a meaningful advancement in the field of emotional TTS synthesis, addressing critical gaps in existing methodologies and datasets.
Methodology
The methodology presented in this paper is robust, focusing on the development of a fine-grained non-verbal expression dataset and a corresponding TTS system. The authors effectively address the limitations of existing datasets by introducing a novel annotation scheme that allows for precise control over non-verbal vocalizations. The use of Grad-TTS as the backbone model, enhanced with an emotion encoder, demonstrates a thoughtful integration of emotional embeddings into the synthesis process. The segmentation and transcription processes are well-detailed, showcasing a clear understanding of audio processing and the importance of high-quality data in training TTS systems.
Results & evaluation
The experimental evaluation is comprehensive, involving subjective assessments of naturalness and emotional expressiveness, as well as emotion recognition accuracy. The use of a diverse set of evaluation metrics, including eMOS and nMOS, provides a nuanced understanding of the model's performance. The results indicate a significant improvement in expressiveness with the fine-grained NV approach, although there is a minor trade-off in perceived naturalness. The emotion-specific analysis adds depth to the findings, illustrating the effectiveness of NV cues in conveying various emotional states.
Reproducibility
The paper provides sufficient detail regarding the dataset construction, model architecture, and evaluation procedures, which enhances reproducibility. However, the absence of a publicly available code repository limits the ability for other researchers to fully replicate the study. The authors could improve reproducibility by sharing their code and trained models.
Limitations
One limitation is the focus on female NV utterances, which may not generalize well to male voices or other demographics. Additionally, the minor trade-off in naturalness when incorporating NVs could be a concern for practical applications. The subjective nature of the evaluations may also introduce variability, as individual preferences for emotional expression can differ widely.
Broader impact
This research has significant implications for the development of more emotionally intelligent conversational AI systems. By enhancing the expressiveness of TTS systems through the integration of non-verbal vocalizations, the work contributes to creating more engaging and human-like interactions in various applications, including virtual assistants, gaming, and mental health support systems. The main contribution of this paper is the introduction of a fine-grained non-verbal expression dataset and a corresponding TTS system that significantly enhances emotional expressiveness in synthesized speech. This work represents a meaningful advancement in the field of emotional TTS synthesis, addressing critical gaps in existing methodologies and datasets.
Author of this paper? Send a correction.