DuplexSLA: A Full-Duplex Spoken Language Model with Synchronized Speech, Language, and Action
Abstract
Recent advances in spoken dialogue language models have shifted from turn-based to full-duplex designs, where the model continuously listens to the user while generating responses. However, existing duplex backbones still lack a native channel for in-conversation planning and tool calling, leaving real-time agentic behaviour either tied to turn boundaries or relegated to an external cascade. We propose DuplexSLA, a native full-duplex Speech-Language-Action foundation model that decodes assistant audio together with a structured action stream on a shared 160 ms chunk timeline. DuplexSLA is built on a dual-stream three-channel formulation: a continuous user audio channel, a discrete assistant audio channel, and a rate-limited textual action channel, all decoded jointly by a single backbone, so that listening, speaking, planning, and tool calling unfold on one shared clock. Two capabilities define the model: (1) semantic-driven turn-taking control, where interruption, pause, and backchannel are handled inside the same backbone instead of by an external semantic VAD; and (2) in-conversation planning and tool calling, where planning text and structured tool calls are emitted on the action channel without halting assistant audio, so that multi-action and backchannel-triggered tool use are interleaved with ongoing speech. To evaluate these capabilities together, we further construct DuplexSLA-Bench, a duplex benchmark covering pause, interrupt, and backchannel turn-taking together with three styles of in-conversation tool calling. Our project page, interactive demos, and the DuplexSLA-Bench evaluation suite are publicly available at https://github.com/hyzhang24/DuplexSLA.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of DuplexSLA, a novel full-duplex Speech-Language-Action model that enables seamless integration of listening, speaking, and action planning in real-time dialogue systems. This work represents a significant step forward in the field of spoken dialogue systems, addressing key limitations of traditional turn-based models and paving the way for more natural and efficient human-computer interactions.
Methodology
The methodology proposed in DuplexSLA is innovative, leveraging a dual-stream three-channel architecture that integrates user audio, assistant audio, and an action channel into a synchronous framework. This design allows for real-time interaction without the latency typically associated with turn-based systems. The paper effectively outlines the architecture and the rationale behind the choice of chunk size and channel design, which are crucial for achieving seamless interaction in spoken dialogue systems. The integration of semantic-driven turn-taking and in-conversation planning is a significant advancement, as it allows for more natural interactions compared to existing models.
Results & evaluation
The experimental evaluation is robust, utilizing the newly created DuplexSLA-Bench to assess the model's performance across various scenarios, including pause, interrupt, and backchannel turn-taking, as well as tool calling. The results demonstrate that DuplexSLA achieves sub-second latency with competitive accuracy compared to traditional ASR + LLM cascades, which is a notable achievement. The comprehensive evaluation metrics, including accuracy and delay, provide a clear picture of the model's performance.
Reproducibility
The paper provides a detailed description of the training process, data construction, and evaluation protocols, which enhances reproducibility. The availability of the project page and code repository further supports efforts to replicate the findings. However, the complexity of the model and the specific training data used may pose challenges for complete reproducibility without access to the same datasets.
Limitations
One limitation of the study is the reliance on a specific chunk size (160 ms), which may not generalize well to all conversational contexts or languages. Additionally, while the model shows promise in handling interruptions and backchannels, its performance in more complex multi-turn dialogues or less structured conversations remains to be evaluated. The paper does not address the potential biases in the training data, which could affect the model's performance in real-world applications.
Broader impact
The implications of DuplexSLA are significant for the development of more interactive and responsive spoken dialogue systems. By enabling real-time planning and tool calling, this model could enhance user experience in various applications, including virtual assistants, customer service bots, and interactive voice response systems. The ability to handle natural conversational phenomena like interruptions and backchannels could lead to more human-like interactions, thereby increasing user satisfaction and engagement. The main contribution of this paper is the introduction of DuplexSLA, a novel full-duplex Speech-Language-Action model that enables seamless integration of listening, speaking, and action planning in real-time dialogue systems. This work represents a significant step forward in the field of spoken dialogue systems, addressing key limitations of traditional turn-based models and paving the way for more natural and efficient human-computer interactions.
Author of this paper? Send a correction.