BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
Abstract
Real-time, full-duplex speech interaction is a key feature of next-generation spoken chatbots, allowing the model to listen and speak at the same time and to handle natural phenomena such as overlap, hesitation, and barge-in. Existing speech language models (SpeechLMs) such as LLaMA-Omni and GLM-4-Voice are still turn-based and rely on an external Voice Activity Detection (VAD) module to mark the end of the user's turn, which fundamentally limits their interactive ability. In this paper, we introduce BayLing-Duplex, a native full-duplex SpeechLM where a single autoregressive LLM decides when to listen, when to speak, and when to stop, with no auxiliary turn-taking module. The design adds only a few special tokens to the standard vocabulary, so it transfers across LLMs and reuses existing training and serving stacks with no architectural adaptation. Starting from the public GLM-4-Voice checkpoint and using only 400K full-duplex samples for fine-tuning followed by a lightweight DPO stage, BayLing-Duplex reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, while improving the speech-response score from 2.17 to 3.39 over Moshi. BayLing-Duplex also matches or surpasses its turn-based counterpart on Llama Questions, Web Questions, and Alpaca-Eval, showing that simultaneous listen-and-speak modeling does not sacrifice response quality.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
BayLing-Duplex presents a novel approach to full-duplex speech interaction using a single autoregressive LLM, significantly enhancing the capabilities of spoken chatbots. The technical contributions, particularly in managing dialogue states and improving interaction quality, position this work as a meaningful advancement in the field of machine learning and speech processing.
Methodology
The methodology presented in BayLing-Duplex is innovative, leveraging a multi-channel interleaved sequence to enable full-duplex speech interaction without the need for an external Voice Activity Detection module. The approach of integrating dialogue-state tokens into the existing GLM-4-Voice architecture is particularly noteworthy, allowing the model to manage turn-taking and interruptions through standard next-token prediction. This design choice minimizes architectural changes while maximizing functionality, which is a significant advancement in the field of speech language models.
Results & evaluation
The experiments are robust, utilizing a well-defined dataset of 400K full-duplex samples for fine-tuning and employing a lightweight Direct Preference Optimization stage. The evaluation metrics are comprehensive, covering turn-taking success, interruption success, and spoken question answering, with results demonstrating significant improvements over existing models. The paper provides clear comparisons with the state-of-the-art model, Moshi, and shows that BayLing-Duplex not only matches but surpasses it in several key metrics.
Reproducibility
The paper outlines the training and evaluation procedures in detail, including the data construction process and the specific configurations used for training. However, the lack of a publicly available demo or project URL limits the ability for others to reproduce the results directly. The reliance on synthesized audio also raises questions about the generalizability of the findings to real-world scenarios.
Limitations
The primary limitations include the reliance on synthesized, single-speaker, and noise-free audio, which may not reflect real-world conditions. The model's performance in noisy environments or with multiple speakers is not evaluated, and the paper acknowledges that further studies are needed to address these aspects. Additionally, the chosen block size introduces a minimum response latency that may not be suitable for all applications.
Broader impact
The implications of this research are significant, as it addresses a critical need for more natural and interactive speech interfaces in various applications, including virtual assistants, customer service bots, and accessibility tools. However, ethical considerations regarding privacy and the potential for misuse in impersonation or disinformation are also highlighted, necessitating careful deployment and monitoring of such technologies. BayLing-Duplex presents a novel approach to full-duplex speech interaction using a single autoregressive LLM, significantly enhancing the capabilities of spoken chatbots. The technical contributions, particularly in managing dialogue states and improving interaction quality, position this work as a meaningful advancement in the field of machine learning and speech processing.
Author of this paper? Send a correction.