Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis
Abstract
While emotional text-to-speech (TTS) has made significant progress, most existing research remains limited to utterance-level emotional expression and fails to support word-level control. Achieving word-level expressive control poses fundamental challenges, primarily due to the complexity of modeling multi-emotion transitions and the scarcity of annotated datasets that capture intra-sentence emotional and prosodic variation. In this paper, we propose WeSCon, the first self-training framework that enables word-level control of both emotion and speaking rate in a pretrained zero-shot TTS model, without relying on datasets containing intra-sentence emotion or speed transitions. Our method introduces a transition-smoothing strategy and a dynamic speed control mechanism to guide the pretrained TTS model in performing word-level expressive synthesis through a multi-round inference process. To further simplify the inference, we incorporate a dynamic emotional attention bias mechanism and fine-tune the model via self-training, thereby activating its ability for word-level expressive control in an end-to-end manner. Experimental results show that WeSCon effectively overcomes data scarcity, achieving state-of-the-art performance in word-level emotional expression control while preserving the strong zero-shot synthesis capabilities of the original TTS model.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of WeSCon, a self-training framework that enables word-level emotional expression control in TTS synthesis, significantly advancing the state-of-the-art in this area. The comprehensive analysis of the methodology, experimental results, and implications highlights its significance in the field of machine learning and speech synthesis.
Methodology
The proposed WeSCon framework introduces a novel two-stage self-training approach that enables word-level emotional expression control in text-to-speech synthesis without the need for extensive emotional datasets. The methodology is well-structured, incorporating a multi-round inference process with transition smoothing and dynamic speed control, which effectively addresses the challenges of emotional transitions in speech synthesis. The dynamic emotional attention bias mechanism further enhances the model's ability to maintain emotional coherence during synthesis. The approach is innovative in its use of self-training to leverage limited data, demonstrating a significant advancement over existing methods that typically rely on extensive emotional datasets.
Results & evaluation
The experimental setup is robust, utilizing both objective and subjective evaluation metrics to assess the performance of WeSCon against strong baseline models. The results indicate that WeSCon achieves state-of-the-art performance in word-level emotional expression control while preserving the zero-shot capabilities of the original TTS model. The ablation studies effectively demonstrate the contributions of each component of the framework, validating the importance of the transition-smoothing and dynamic speed control mechanisms.
Reproducibility
The paper provides sufficient details regarding the training setup, model architecture, and evaluation metrics, which should facilitate reproducibility. However, the reliance on specific datasets and the absence of a publicly available implementation may hinder full reproducibility for some researchers.
Limitations
The paper acknowledges limitations, including the lack of gradual emotion transitions and the fixed set of discrete emotions, which restricts the model's expressiveness. Additionally, the model's reliance on predefined emotional transitions may limit its adaptability in dynamic contexts. Future work is suggested to explore more flexible, context-aware control strategies.
Broader impact
The WeSCon framework has potential applications in various domains, including virtual agents, emotional storytelling, and expressive speech synthesis. However, it also raises ethical concerns related to speaker impersonation and the potential for generating biased or inappropriate outputs, necessitating careful consideration in deployment. The main contribution of this paper is the introduction of WeSCon, a self-training framework that enables word-level emotional expression control in TTS synthesis, significantly advancing the state-of-the-art in this area. The comprehensive analysis of the methodology, experimental results, and implications highlights its significance in the field of machine learning and speech synthesis.
Author of this paper? Send a correction.