Semantic-Aware Interruption Detection in Spoken Dialogue Systems: Benchmark, Metric, and Model
Abstract
Achieving natural full-duplex interaction in spoken dialogue systems (SDS) remains a challenge due to the difficulty of accurately detecting user interruptions. Current solutions are polarized between "trigger-happy" VAD-based methods that misinterpret backchannels and robust end-to-end models that exhibit unacceptable response delays. Moreover, the absence of real-world benchmarks and holistic metrics hinders progress in the field. This paper presents a comprehensive frame-work to overcome these limitations. We first introduce SID-Bench, the first benchmark for semantic-aware interruption detection built entirely from real-world human dialogues. To provide a rigorous assessment of the responsiveness-robustness trade-off, we propose the Average Penalty Time (APT) metric, which assigns a temporal cost to both false alarms and late responses. Building on this framework, we design an LLM-based detection model optimized through a novel training paradigm to capture subtle semantic cues of intent. Experimental results show that our model significantly outperforms mainstream baselines, achieving a nearly threefold reduction in APT. By successfully resolving the long-standing tension between speed and stability, our work establishes a new state-of-the-art for intelligent interruption handling in SDS. To facilitate future research, SID-Bench and the associated code are available at: https://github.com/xkx-hub/SID-bench.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of SID-Bench and the APT metric, which together provide a comprehensive framework for evaluating and improving interruption detection in spoken dialogue systems. This work significantly enhances the understanding of user interruptions in conversational AI, offering a robust methodology and strong experimental validation that addresses key challenges in the field.
Methodology
The paper introduces a novel framework for interruption detection in spoken dialogue systems, emphasizing the creation of SID-Bench, a benchmark based on real-world data, and the Average Penalty Time (APT) metric for evaluation. The methodology is robust, incorporating a two-stage training paradigm that effectively leverages large language models (LLMs) for semantic understanding, and a hybrid annotation approach that combines LLMs with forced alignment for precise interruption labeling. This innovative approach addresses the limitations of existing VAD-based systems and enhances the model's ability to discern genuine interruptions from backchannels.
Results & evaluation
The experimental results are comprehensive, demonstrating a significant performance improvement in the proposed model over existing baselines across various metrics, including APT, FIR, and IRL. The use of SID-Bench allows for a rigorous evaluation of the model's capabilities in real-world scenarios, and the results clearly illustrate the trade-off between responsiveness and robustness, validating the effectiveness of the proposed methods.
Reproducibility
The paper provides sufficient details regarding the model architecture, training procedures, and evaluation metrics, which would allow for reproducibility. The availability of SID-Bench and the associated code on GitHub further enhances the potential for other researchers to replicate the study and build upon the findings.
Limitations
One limitation is the reliance on a specific set of conversational data, which may not encompass all possible interaction scenarios. Additionally, while the model achieves significant improvements in APT, further exploration into its performance across diverse languages and dialects could be beneficial. The paper also does not address the computational resources required for training the LLM-based model, which may limit accessibility for some researchers.
Broader impact
The proposed framework and benchmark have the potential to significantly advance the field of spoken dialogue systems by providing a more nuanced understanding of interruption handling. This could lead to more natural and efficient human-computer interactions, with applications in customer service, virtual assistants, and other conversational AI systems. The introduction of SID-Bench sets a precedent for future research in this area, encouraging the development of more sophisticated models that can better understand human intent. The main contribution of this paper is the introduction of SID-Bench and the APT metric, which together provide a comprehensive framework for evaluating and improving interruption detection in spoken dialogue systems. This work significantly enhances the understanding of user interruptions in conversational AI, offering a robust methodology and strong experimental validation that addresses key challenges in the field.
Author of this paper? Send a correction.