CoDeTT: A Context-Aware Decision Benchmark for Turn-Taking Evaluation
Abstract
Turn-taking modeling is fundamental to spoken dialogue systems, yet its evaluation remains fragmented and often limited to binary boundary detection under narrow interaction settings. Such protocols hinder systematic comparison and obscure model weaknesses across conversational conditions. We present CoDeTT, a context-aware decision benchmark for turn-taking evaluation. CoDeTT formulates turn-taking as a structured decision problem and constructs a multi-scenario dataset with fine-grained decision categories and controlled context variations. Under a unified evaluation protocol, we assess representative existing models and observe substantial performance disparities across decision types and interaction scenarios. CoDeTT provides a standardized benchmark for systematic and context-aware evaluation of turn-taking systems. The benchmark dataset and evaluation toolkit are available at https://github.com/YingaoWang-casia/CoDeTT.github.io.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of CoDeTT, a context-aware decision benchmark for turn-taking evaluation that systematically addresses the limitations of existing evaluation protocols. This work is significant as it enhances the understanding of model performance in conversational systems and provides a foundation for future research in turn-taking dynamics.
Methodology
The proposed methodology introduces CoDeTT, a structured decision benchmark for turn-taking evaluation that effectively captures the complexities of conversational interactions. By formulating turn-taking as a structured decision problem and creating a multi-scenario dataset with fine-grained decision categories, the authors provide a robust framework for evaluating existing models. The use of a hierarchical taxonomy and a Two-Stage Funnel Evaluation Protocol enhances the depth of analysis, allowing for both coarse and fine-grained assessments of model performance.
Results & evaluation
The experiments are comprehensive, utilizing a large bilingual dataset of over 300 hours of dialogue, which is well-annotated and balanced across various decision scenarios. The evaluation of existing models under the CoDeTT benchmark reveals significant performance disparities, highlighting the utility of the benchmark in exposing model weaknesses. The introduction of the Semantic Misalignment Rate (SMR) as a diagnostic metric is particularly noteworthy, as it provides insights into the underlying reasoning of models.
Reproducibility
The paper provides a clear description of the dataset construction process and the evaluation protocol, which enhances reproducibility. The availability of the dataset and evaluation toolkit on GitHub further supports this aspect. However, the paper could benefit from more detailed implementation guidelines for replicating the experiments.
Limitations
One limitation is the reliance on synthetic data generation for part of the dataset, which may introduce biases or artifacts that could affect model performance. Additionally, while the benchmark exposes decision-specific performance variations, it may not fully account for all contextual nuances present in real-world conversations.
Broader impact
The CoDeTT benchmark has the potential to significantly influence the development of more sophisticated and context-aware conversational agents, improving user experience in spoken dialogue systems. By providing a standardized evaluation framework, it encourages further research into turn-taking modeling and may lead to advancements in human-computer interaction. The main contribution of this paper is the introduction of CoDeTT, a context-aware decision benchmark for turn-taking evaluation that systematically addresses the limitations of existing evaluation protocols. This work is significant as it enhances the understanding of model performance in conversational systems and provides a foundation for future research in turn-taking dynamics.
Author of this paper? Send a correction.