EmoTra-TTS: Smooth Intra-Utterance Emotion Transitions for Speech Synthesis
Abstract
Psychological research on emotion dynamics has established that human affect is a continuous, evolving process: emotions rise, decay, and transition within seconds. Current emotional text-to-speech (TTS) systems, however, condition on a single discrete label or static embedding per utterance, fundamentally misaligning with the temporal nature of affect. While recent LLM-based TTS systems may implicitly vary prosody through text understanding, such variation is neither explicitly controllable nor precise enough for targeted intra-utterance transitions. We address three challenges: (1) a multi-pass flow blending pipeline synthesizes frame-aligned transition audio, circumventing the scarcity of natural intra-utterance transitions; (2) dual-stage Valence-Arousal-Dominance (VAD) conditioning guides prosodic planning in the LLM and acoustic realization in the flow decoder via frame-level VAD embeddings; (3) direction-magnitude decoupled injection structurally separates emotion direction from injection magnitude, preventing content degradation. EmoTra-TTS adds only +0.43% parameters with no latency overhead, achieves 30%-87% relative improvement on emotion transition quality, corroborated by 64.4%-79.5% overall win rates in pairwise preference tests against four SOTA baselines and two commercial systems.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
EmoTra-TTS makes a significant contribution to expressive TTS by introducing a novel framework for smooth intra-utterance emotion transitions, combining synthetic data generation with a robust dual-stage conditioning mechanism. The rigorous evaluation and clear ablation studies demonstrate the effectiveness of the proposed methods, although generalization beyond the training dataset remains an area for future work.
Methodology
The paper addresses the underexplored problem of intra-utterance emotion transitions in TTS, moving beyond static emotion conditioning. The proposed EmoTra-TTS framework is built on three pillars: (1) a synthetic data generation pipeline using multi-pass flow blending to create frame-aligned transition samples, addressing data scarcity; (2) dual-stage conditioning where an LLM handles prosodic planning via temporal VAD tokens, and a flow decoder realizes acoustic features via frame-level VAD embeddings; and (3) a direction-magnitude decoupled injection mechanism using LayerNorm and a fixed scale to prevent content degradation when injecting emotion signals into a frozen decoder. The methodology is sound and addresses specific failure modes of previous approaches (e.g., content collapse, magnitude mismatch) with rigorous ablation studies. The separation of prosodic planning and acoustic realization is a logical extension of current LLM-based TTS architectures.
Results & evaluation
The experimental evaluation is comprehensive, including MOS-based metrics (Quality, Emotion, Transition), speaker similarity, WER, and pairwise preference tests against four open-source baselines and two commercial systems. The results show significant improvements in transition smoothness (MOS-Tra) and overall preference. The inclusion of an objective metric (Prosodic Jerk Ratio) strengthens the findings. The ablation studies effectively isolate the contributions of each component. However, the evaluation is limited to the EmoVoice-DB dataset, which restricts claims about generalization to other speakers or languages. The use of synthetic data for training, while effective, introduces a domain gap that is partially mitigated by the quality filtering but remains a limitation.
Reproducibility
The paper provides detailed descriptions of the architecture, training objectives, and hyperparameters. The code and demo are publicly available. The synthetic data generation pipeline is clearly described, allowing for reproduction of the training data. The use of a frozen backbone (CosyVoice2) and specific injection mechanisms makes the implementation straightforward to replicate.
Limitations
The primary limitation is the reliance on the EmoVoice-DB dataset, which has limited speaker diversity and language coverage. The paper acknowledges that zero-shot generalization to unseen speakers is not demonstrated. Additionally, the emotion control stability for subtle VAD differences is noted as inconsistent. The piecewise linear interpolation of VAD trajectories is a simplification of natural emotion dynamics.
Broader impact
EmoTra-TTS contributes to the field of affective computing and expressive speech synthesis. By enabling smooth, controllable emotion transitions, it opens up new possibilities for storytelling, dubbing, and interactive virtual agents. The ethical statement addresses potential misuse, such as generating deceptive audio, and emphasizes responsible use. The work encourages further research into dynamic emotion modeling in speech. EmoTra-TTS makes a significant contribution to expressive TTS by introducing a novel framework for smooth intra-utterance emotion transitions, combining synthetic data generation with a robust dual-stage conditioning mechanism. The rigorous evaluation and clear ablation studies demonstrate the effectiveness of the proposed methods, although generalization beyond the training dataset remains an area for future work.
Author of this paper? Send a correction.