VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs
Abstract
Speech Large Language Models (LLMs) show great promise for speech emotion recognition (SER) via generative interfaces. However, shifting from closed-set classification to open text generation introduces zero-shot stochasticity, making evaluation highly sensitive to prompts. Additionally, conventional speech LLMs benchmarks overlook the inherent ambiguity of human emotion. Hence, we present VoxEmo, a comprehensive SER benchmark encompassing 35 emotion corpora across 15 languages for Speech LLMs. VoxEmo provides a standardized toolkit featuring varying prompt complexities, from direct classification to paralinguistic reasoning. To reflect real-world perception/application, we introduce a distribution-aware soft-label protocol and a prompt-ensemble strategy that emulates annotator disagreement. Experiments reveal that while zero-shot speech LLMs trail supervised baselines in hard-label accuracy, they uniquely align with human subjective distributions.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of VoxEmo, a comprehensive benchmarking framework for speech emotion recognition that addresses the challenges of prompt sensitivity and human emotion ambiguity in the evaluation of speech LLMs. The technical contributions, including the standardized toolkit and innovative evaluation strategies, position this work as a significant advancement in the field of SER.
Methodology
The paper introduces VoxEmo, a novel benchmarking framework for speech emotion recognition (SER) using speech LLMs. The methodology is well-structured, addressing the challenges of prompt sensitivity and human emotion ambiguity through a comprehensive toolkit that includes a distribution-aware soft-label protocol and a prompt-ensemble strategy. The approach of utilizing multiple prompts to capture the stochastic nature of LLM outputs is innovative, although it may lead to increased complexity in evaluation.
Results & evaluation
The experiments are extensive, covering 35 emotion corpora across 15 languages. The results demonstrate the performance of two speech LLMs (Qwen2-Audio and Audio Flamingo) under various prompt configurations. The analysis of zero-shot performance and the impact of supervised fine-tuning is thorough, providing valuable insights into the strengths and weaknesses of the models. However, the paper could benefit from more detailed comparisons with existing state-of-the-art methods.
Reproducibility
The paper emphasizes reproducibility by providing a standardized evaluation toolkit and clear descriptions of the experimental setup, including the selection of models and evaluation metrics. However, the reliance on specific models and the absence of a public code repository may hinder full reproducibility.
Limitations
The paper acknowledges several limitations, including the focus on only two models with the same audio encoder, the potential for hyperparameter mismatch during fine-tuning, and the restriction of soft-label evaluation to a limited number of datasets. Additionally, the study does not explore within-dataset factors that could affect performance.
Broader impact
The proposed benchmark has significant implications for the development of affect-aware systems in human-computer interaction and speech analytics. By addressing the ambiguity of human emotion and providing a framework for evaluating generative models, this work could lead to advancements in more nuanced and effective emotion recognition systems. The main contribution of this paper is the introduction of VoxEmo, a comprehensive benchmarking framework for speech emotion recognition that addresses the challenges of prompt sensitivity and human emotion ambiguity in the evaluation of speech LLMs. The technical contributions, including the standardized toolkit and innovative evaluation strategies, position this work as a significant advancement in the field of SER.
Author of this paper? Send a correction.