The ICASSP 2026 HumDial Challenge: Benchmarking Human-like Spoken Dialogue Systems in the LLM Era
Abstract
Driven by the rapid advancement of Large Language Models (LLMs), particularly Audio-LLMs and Omni-models, spoken dialogue systems have evolved significantly, progressively narrowing the gap between human-machine and human-human interactions. Achieving truly ``human-like'' communication necessitates a dual capability: emotional intelligence to perceive and resonate with users' emotional states, and robust interaction mechanisms to navigate the dynamic, natural flow of conversation, such as real-time turn-taking. Therefore, we launched the first Human-like Spoken Dialogue Systems Challenge (HumDial) at ICASSP 2026 to benchmark these dual capabilities. Anchored by a sizable dataset derived from authentic human conversations, this initiative establishes a fair evaluation platform across two tracks: (1) Emotional Intelligence, targeting long-term emotion understanding and empathetic generation; and (2) Full-Duplex Interaction, systematically evaluating real-time decision-making under `` listening-while-speaking'' conditions. This paper summarizes the dataset, track configurations, and the final results.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the establishment of the HumDial Challenge, which benchmarks emotional intelligence and full-duplex interaction in spoken dialogue systems. This initiative represents a significant step forward in evaluating and improving the capabilities of dialogue systems, particularly in the context of LLM advancements, and sets a foundation for future research in human-computer interaction.
Methodology
The paper presents a well-structured methodology for assessing human-like spoken dialogue systems through the HumDial Challenge, focusing on two critical aspects: Emotional Intelligence and Full-Duplex Interaction. The use of a hybrid dataset combining LLM-generated scripts and human performance is innovative and addresses the limitations of existing benchmarks. The evaluation framework, which includes both automated and human assessments, adds rigor to the methodology. However, the reliance on professional actors for data collection may introduce biases that need to be acknowledged.
Results & evaluation
The results section highlights the challenge's competitive nature, with over 100 registered teams and 15 valid submissions. The performance metrics indicate that while teams excelled in emotional tracking and reasoning, generating empathetic responses remains a challenge. The detailed breakdown of scores across tasks provides insight into the strengths and weaknesses of various systems, emphasizing the need for further advancements in real-time interaction capabilities.
Reproducibility
The paper outlines the evaluation methodology and dataset construction processes in detail, which aids reproducibility. However, the absence of specific implementation details or code availability limits the ability for others to replicate the results fully. The authors should consider providing access to the datasets and evaluation scripts to enhance reproducibility.
Limitations
One identified limitation is the potential bias introduced by using professional actors for the dataset, which may not fully capture the variability of real-world interactions. Additionally, the challenge's focus on specific emotional dynamics and interaction scenarios may not encompass the full spectrum of human dialogue complexities.
Broader impact
The HumDial Challenge has significant implications for the development of more sophisticated spoken dialogue systems that can engage users in a more human-like manner. The focus on emotional intelligence and real-time interaction is particularly relevant for applications in mental health, customer service, and social robotics, where empathetic communication is crucial. The main contribution of this paper is the establishment of the HumDial Challenge, which benchmarks emotional intelligence and full-duplex interaction in spoken dialogue systems. This initiative represents a significant step forward in evaluating and improving the capabilities of dialogue systems, particularly in the context of LLM advancements, and sets a foundation for future research in human-computer interaction.
Author of this paper? Send a correction.