HumDial-EIBench: A Human-Recorded Multi-Turn Emotional Intelligence Benchmark for Audio Language Models
Abstract
Evaluating the emotional intelligence (EI) of audio language models (ALMs) is critical. However, existing benchmarks mostly rely on synthesized speech, are limited to single-turn interactions, and depend heavily on open-ended scoring. This paper proposes HumDial-EIBench, a comprehensive benchmark for evaluating ALMs' EI. Using real-recorded human dialogues from the ICASSP 2026 HumDial Challenge, it reformulates emotional tracking and causal reasoning into multiple-choice questions with adversarial distractors, mitigating subjective scoring bias for cognitive tasks. It retains the generation of empathetic responses and introduces an acoustic-semantic conflict task to assess robustness against contradictory multimodal signals. Evaluations of eight ALMs reveal that most models struggle with multi-turn emotional tracking and implicit causal reasoning. Furthermore, all models exhibit decoupled textual and acoustic empathy, alongside a severe text-dominance bias during cross-modal conflicts.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces HumDial-EIBench, a novel benchmark for evaluating the emotional intelligence of audio language models using real human dialogues. The comprehensive methodology and significant findings regarding model performance gaps contribute meaningfully to the advancement of multimodal AI systems, highlighting the need for improved emotional understanding in AI interactions.
Methodology
The proposed methodology is robust, leveraging real human dialogues to create a comprehensive benchmark for evaluating emotional intelligence in audio language models. The reformulation of tasks into multiple-choice questions with adversarial distractors is innovative and addresses the subjective biases present in previous benchmarks. The introduction of an acoustic-semantic conflict task is particularly noteworthy, as it evaluates models' abilities to handle contradictory multimodal signals, which is a significant gap in existing frameworks. The structured data construction pipeline ensures high-quality recordings and a controlled evaluation environment, enhancing the reliability of the results.
Results & evaluation
The experiments conducted on eight state-of-the-art audio language models provide valuable insights into their performance across various emotional intelligence tasks. The results highlight critical deficiencies in current models, particularly in multi-turn emotional tracking and implicit causal reasoning. The use of both automated and human evaluations for different tasks adds depth to the analysis, although the reliance on LLMs for some scoring introduces variability. The findings are well-supported by quantitative metrics and qualitative assessments, making a strong case for the proposed benchmark's effectiveness.
Reproducibility
The paper provides a clear description of the methodology and evaluation metrics, along with a link to the GitHub repository for accessing the benchmark. However, details on the specific implementations of the evaluated models and their configurations are limited, which may hinder full reproducibility. The authors could enhance this aspect by providing more granular information on the experimental setup and model parameters.
Limitations
The study acknowledges limitations, such as the high variance in text empathy evaluation scores, indicating challenges in objectively quantifying empathy depth. Additionally, the acoustic-semantic conflict evaluation is currently limited to single-turn utterances, which may not fully capture the complexities of real-world interactions. Future work is needed to expand multi-turn conflict scenarios and improve automatic evaluation metrics.
Broader impact
The development of HumDial-EIBench has significant implications for the field of emotional intelligence in AI, particularly in enhancing the capabilities of audio language models. By addressing critical gaps in existing benchmarks, this work paves the way for more nuanced evaluations of multimodal systems, potentially leading to advancements in applications such as conversational agents, mental health support systems, and interactive entertainment. This paper introduces HumDial-EIBench, a novel benchmark for evaluating the emotional intelligence of audio language models using real human dialogues. The comprehensive methodology and significant findings regarding model performance gaps contribute meaningfully to the advancement of multimodal AI systems, highlighting the need for improved emotional understanding in AI interactions.
Author of this paper? Send a correction.