DiffuPlex: Accelerating Full-Duplex Spoken Dialog Models via Rolling Masked Diffusion
Abstract
Recent full-duplex spoken dialog models enable simultaneous listening and speaking, but fine-grained models still advance their backbone autoregressively at every interaction frame. We introduce DiffuPlex, a rolling masked diffusion framework that reduces this sequential computation by predicting multiple future user and assistant frames in a single backbone wake. DiffuPlex consumes only a confident prefix of each predicted future while interaction continues at the original frame rate. As user speech arrives, it checks the corresponding user predictions and, when the interaction diverges, preserves already played assistant content while revising only the unplayed future. We consider two inference policies over the same predictor: DiffuPlex-LISTEN consumes multiple future frames when they predict assistant silence, whereas DiffuPlex-SPEAK can also consume predicted assistant speech. Across full-duplex interaction and spoken-language evaluations, DiffuPlex substantially reduces sequential backbone computation while largely preserving interaction behavior and general capability. DiffuPlex-LISTEN and DiffuPlex-SPEAK achieve $1.46\times$ and $1.59\times$ deployment-path wall-clock speedups and $1.61\times$ and $1.80\times$ Core LM speedups, with all measured backbone invocations completing within the 80ms interaction interval. Human evaluation shows that LISTEN preserves speech naturalness and conversational quality, while SPEAK retains conversational quality with some degradation in speech naturalness.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
[The paper introduces DiffuPlex, a rolling masked diffusion framework that accelerates full-duplex spoken dialog models by predicting multiple future interaction frames in a single backbone pass, achieving significant computational speedups while preserving interaction quality through careful confidence-based retention and revision mechanisms.].
Methodology
The paper proposes DiffuPlex, a "rolling masked diffusion" framework to accelerate full-duplex spoken dialog models. The core innovation is replacing the standard autoregressive step (one backbone forward pass per 80ms frame) with a mechanism that predicts a block of future user and assistant frames in a single "backbone wake." It uses learned MASK tokens for unavailable future inputs and trains via self-distillation from a frozen autoregressive backbone using LoRA adapters. The inference process involves predicting a future block, consuming a confident prefix of the assistant's speech, and checking user predictions against actual incoming audio. If a divergence is detected, it revises only the unplayed future. This approach is technically sound and addresses a real computational bottleneck in fine-grained full-duplex models. The distinction between "Listen" (consuming only silence predictions) and "Speak" (consuming speech predictions) policies is a practical and well-motivated design choice.
Results & evaluation
The experiments are comprehensive, covering full-duplex interaction quality (Full-Duplex-Bench), general spoken language capability (VoiceBench, OpenAudioBench), human evaluation (NMOS/DMOS), and inference efficiency (wall-clock and Core LM speedups). The results show significant speedups (1.46x-1.59x wall-clock) while largely preserving interaction quality. The analysis of why "Speak" mode degrades speech naturalness (loss of same-frame semantic conditioning for acoustic codebooks) is insightful and demonstrates a deep understanding of the underlying architecture. The comparison against deferred-execution baselines (skipping frames and emitting silence) effectively highlights the benefit of predicting the future rather than just skipping computation.
Reproducibility
The paper provides detailed descriptions of the training layout, masking schedules, and inference policies. It specifies the backbone used (personaplex-rl-seamless) and the dataset (Seamless Interaction). While code release is planned post-review, the level of detail in the appendices (referenced but not fully provided in the text snippet, though implied to be extensive) suggests good reproducibility. The use of standard components (LoRA, Mimi codebooks) aids in understanding the implementation.
Limitations
The method is currently adapted to a specific backbone (PersonaPlex family) and may not transfer automatically to other architectures, as noted in the limitations section. The "Speak" policy suffers from speech quality degradation due to architectural constraints (missing same-frame conditioning), which limits its practical utility compared to the "Listen" policy. The speedups, while significant, are moderate (around 1.5x), and the method adds complexity to the inference pipeline.
Broader impact
This work contributes to the efficiency of real-time speech interaction systems, which is crucial for deploying LLMs in conversational agents. By reducing the computational cost of full-duplex models, it makes such advanced interaction styles more feasible on edge devices or in high-concurrency server environments. The insights into the trade-offs between prediction confidence, revision, and speech quality are valuable for the broader community working on speech LLMs. [The paper introduces DiffuPlex, a rolling masked diffusion framework that accelerates full-duplex spoken dialog models by predicting multiple future interaction frames in a single backbone pass, achieving significant computational speedups while preserving interaction quality through careful confidence-based retention and revision mechanisms.].
Author of this paper? Send a correction.