AffectSpeech: A Large-Scale Emotional Speech Dataset with Fine-Grained Textual Descriptions for Speech Emotion Captioning and Synthesis

Tianhua Qi, Wenming Zheng, ★ Björn W. Schuller ...Full author list in the original paper.

Southeast University +5 · arXiv

Affiliations: Southeast University, Shenzhen Loop Area Institute, Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Technical University of Munich, Imperial College London

Paper GitHub Demo

Abstract

Emotion is essential in spoken communication, yet most existing frameworks in speech emotion modeling rely on predefined categories or low-dimensional continuous attributes, which offer limited expressive capacity. Recent advances in speech emotion captioning and synthesis have shown that textual descriptions provide a more flexible and interpretable alternative for representing affective characteristics in speech. However, progress in this direction is hindered by the lack of an emotional speech dataset aligned with reliable and fine-grained natural language annotations. To tackle this, we introduce AffectSpeech, a large-scale corpus of human-recorded speech enriched with structured descriptions for fine-grained emotion analysis and generation. Each utterance is characterized across six complementary dimensions, including sentiment polarity, open-vocabulary emotion captions, intensity level, prosodic attributes, prominent segments, and semantic content, enabling multi-granular modeling of vocal expression. To balance annotation quality and scalability, we adopt a human-LLM collaborative annotation pipeline that integrates algorithmic pre-labeling, multi-LLM description generation, and human-in-the-loop verification. Furthermore, these annotations are reformulated into diverse descriptive styles to enhance linguistic diversity and reduce stylistic bias in downstream modeling. Experimental results on speech emotion captioning and synthesis demonstrate that models trained on AffectSpeech consistently achieve superior performance across multiple evaluation settings.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The paper introduces AffectSpeech, a large-scale emotional speech dataset with fine-grained textual descriptions, significantly advancing the field of speech emotion analysis and synthesis. The innovative methodology, comprehensive experimental evaluation, and potential for broader applications underscore its importance in enhancing machine understanding of human emotions in speech.

Methodology

The methodology presented in the paper is robust and innovative, particularly in the use of a human-LLM collaborative annotation pipeline, which integrates algorithmic pre-labeling, multi-LLM description generation, and human-in-the-loop verification. This approach not only enhances the quality of the annotations but also ensures scalability, addressing a significant gap in existing datasets for speech emotion analysis. The multi-granular annotation across six dimensions allows for a comprehensive representation of emotional states, which is a notable advancement over traditional categorical or dimensional approaches. The incorporation of diverse descriptive styles further enriches the dataset, promoting linguistic diversity and reducing stylistic bias.

Results & evaluation

The experimental evaluation is thorough, with extensive benchmarks demonstrating the effectiveness of the AffectSpeech dataset in improving performance in speech emotion captioning and synthesis tasks. The paper provides both objective metrics (emotion accuracy, prosody accuracy, description quality) and subjective evaluations (human preference tests), which collectively validate the dataset's utility. The results indicate significant improvements over existing datasets, showcasing the dataset's ability to facilitate nuanced emotional understanding and synthesis.

Reproducibility

The paper provides a clear description of the data collection and annotation processes, along with the experimental setups used for fine-tuning models. However, while the methodology is well-documented, the actual code and models used for the experiments are not provided in the paper, which may hinder full reproducibility. The authors do link to a GitHub repository, which is a positive step towards enabling others to replicate their work.

Limitations

One limitation of the study is the reliance on existing datasets for the construction of AffectSpeech, which could introduce biases inherent in those datasets. Additionally, while the human-LLM collaborative approach enhances annotation quality, it may still be susceptible to the limitations of LLMs, such as potential hallucinations or inaccuracies in generated descriptions. The dataset is currently limited to English, which may restrict its applicability in multilingual contexts.

Broader impact

The AffectSpeech dataset has significant potential applications in various fields, including affective computing, human-computer interaction, and conversational AI. By providing a more nuanced understanding of emotional expression in speech, it can enhance the development of empathetic conversational agents and improve user experiences in interactive systems. The dataset could also serve as a foundation for future research in multilingual emotion recognition and synthesis. The paper introduces AffectSpeech, a large-scale emotional speech dataset with fine-grained textual descriptions, significantly advancing the field of speech emotion analysis and synthesis. The innovative methodology, comprehensive experimental evaluation, and potential for broader applications underscore its importance in enhancing machine understanding of human emotions in speech.

Author of this paper? Send a correction.