VoxMind: An End-to-End Agentic Spoken Dialogue System
Abstract
Recent end-to-end spoken dialogue models enable natural interaction. However, as user demands become increasingly complex, models that rely solely on conversational abilities often struggle to cope. Incorporating agentic capabilities is therefore essential: by enabling tool use, these models can extend their knowledge boundaries and better solve real-world tasks. Yet, existing research has largely concentrated on core perception and generation, with comparatively limited exploration of such tool-augmented extensions. To bridge this gap, we present VoxMind, an integrated framework designed to equip end-to-end spoken dialogue models with comprehensive agentic abilities. Leveraging our curated 470-hour AgentChat dataset, we incorporate a "Think-before-Speak" mechanism, enabling the model to internalize structured reasoning as a critical prerequisite for planning and response generation. Furthermore, to mitigate latency bottlenecks caused by large-scale tool integration, we propose a Multi-Agent Dynamic Tool Management architecture. By asynchronously delegating retrieval tasks to an auxiliary agent aligned with the main model's reasoning trajectory, this system effectively decouples inference latency from toolset size. Experimental results confirm that VoxMind achieves significant improvements in agent performance: compared with strong baselines, the task completion rate increases from 34.88% to 74.57%, outperforming Gemini-2.5-Pro on spoken agent tasks while preserving general conversational quality. The source code and associated data are publicly available at https://github.com/MM-Speech/VoxMind.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this work is the introduction of VoxMind, a novel framework that enhances spoken dialogue systems with agentic capabilities through structured reasoning and dynamic tool management. This paper significantly advances the field by addressing critical gaps in the capabilities of existing end-to-end spoken dialogue models, providing a robust theoretical and practical foundation for future research and applications.
Methodology
The paper presents a well-structured and innovative methodology for developing an end-to-end spoken dialogue system, VoxMind, which integrates agentic capabilities through a "Think-before-Speak" mechanism and a Multi-Agent Dynamic Tool Management architecture. The formal definition of End-to-End Spoken Agents and the construction of the AgentChat dataset are significant contributions that address existing gaps in the field. The proposed methods are theoretically sound and practically relevant, demonstrating a clear understanding of the challenges in spoken dialogue systems.
Results & evaluation
The experiments are comprehensive, comparing VoxMind against strong baselines, including closed-source models. The reported improvements in task completion rates and core agent competencies are substantial, showcasing the effectiveness of the proposed framework. The evaluation metrics used are appropriate, and the ablation studies provide insights into the importance of reasoning capabilities in enhancing performance.
Reproducibility
The paper provides sufficient implementation details, including training configurations and dataset compositions, which facilitate reproducibility. The source code and dataset are publicly available, further supporting the reproducibility of the results.
Limitations
The paper acknowledges the inherent latency introduced by the "Think-before-Speak" mechanism and the potential limitations of the AgentChat dataset, which may not fully capture the nuances of spontaneous spoken language. Future work should address these issues to enhance the practical applicability of the system.
Broader impact
The advancements presented in VoxMind have significant implications for real-world applications in spoken dialogue systems, particularly in areas requiring complex reasoning and tool usage. The integration of agentic capabilities could enhance user interactions in various domains, including customer service, education, and personal assistance. The main contribution of this work is the introduction of VoxMind, a novel framework that enhances spoken dialogue systems with agentic capabilities through structured reasoning and dynamic tool management. This paper significantly advances the field by addressing critical gaps in the capabilities of existing end-to-end spoken dialogue models, providing a robust theoretical and practical foundation for future research and applications.
Author of this paper? Send a correction.