A Harness for Synthesizing Diverse Naturalistic Full-Duplex Conversations
Abstract
Full-duplex dialogue systems, which listen while speaking, must distinguish a completed turn from a pause within a turn and an interruption that requests a turn from a brief acknowledgment or speech addressed to a third party. Yet existing conversational corpora provide limited control over these events and limited labels for their intent. We present a pipeline for synthesizing intent-labeled, two-channel conversational speech from relational event lists. An LLM authors each event's speaker, text, conversational act, and attachment to an earlier event without predicting absolute timestamps. Events are synthesized independently, aligned with their source text, and placed on a shared clock, so turn-taking landmarks are measured from the rendered signal while silence durations are specified or sampled from turn-taking distributions. The pipeline covers 42 phenomena across eight families in English and Mandarin, derives frame-level system actions from authored intent, and promotes diversity using small, diverse sets of prior examples and batch prompts that request alternatives with self-reported probabilities. Ablations show gains in each targeted diversity dimension. On a four-action label space for taking, holding, releasing, and not holding the conversational floor, a semantic voice-activity detector using only current and past audio reaches start-speaking and start-listening F1 scores of 0.819 and 0.802. When generating its own responses, the full-duplex speech model Moshi takes 0.85 of the reference turns after fine-tuning on the generated corpus, compared with 0.44 before fine-tuning. Its frame-level precision for predicting system-floor occupancy rises from 0.46 to 0.88. With reference context at each step, its frame-level floor F1 rises from 0.893 to 0.962. These results show that controlled synthesis can provide learnable and transferable supervision for full-duplex turn management.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel pipeline for synthesizing intent-labeled, full-duplex conversational speech by decoupling LLM-based semantic authoring from acoustic realization via forced alignment. This approach effectively addresses the supervision gap in existing corpora by providing precise, intent-aware labels for turn-taking events, leading to significant improvements in the performance of full-duplex speech models like Moshi in managing conversational floor occupancy.
Methodology
The paper proposes a robust pipeline for synthesizing full-duplex conversational data by decoupling semantic authoring from acoustic realization. The core innovation is the "relational event representation," where an LLM defines conversational acts (turn, barge-in, backchannel) and their temporal relationships to previous events without predicting absolute timestamps. This is a significant methodological improvement over previous approaches that attempted to have LLMs predict precise timing, which is unreliable. The pipeline uses forced alignment to measure actual speech boundaries and inserts silence based on turn-taking distributions, ensuring that the acoustic timing is natural while the intent labels remain precise. The use of "diversity hubs" to prevent mode collapse in LLM generation is a clever application of negative memory. The separation of intent (semantic) from activity (acoustic) allows for the creation of frame-level labels that distinguish between a barge-in (which should stop the system) and a backchannel (which should not), a critical distinction for full-duplex models that voice activity detection (VAD) alone cannot provide.
Results & evaluation
The evaluation is strong and directly addresses the paper's claims. The authors train a semantic voice-activity detector and fine-tune the Moshi full-duplex speech model on the generated corpus. The results show significant improvements: Moshi's ability to take the correct turn increases from 0.44 to 0.85, and frame-level floor occupancy precision rises from 0.46 to 0.88. These metrics are highly relevant to the task of turn-taking management. The ablation studies on diversity mechanisms further validate the pipeline's components. The use of both English and Mandarin data demonstrates the pipeline's cross-lingual applicability.
Reproducibility
The paper provides detailed descriptions of the pipeline stages, including the specific models used (DeepSeek-V4-Pro for authoring, Qwen3-TTS for synthesis, Qwen3-ForcedAligner for alignment). The mathematical formulation for timeline assembly is clear. However, the specific hyperparameters for the turn-taking distributions and the exact prompt templates are referenced in appendices that are not fully visible in the provided text, which slightly limits immediate reproducibility. The reliance on specific commercial/proprietary LLMs (DeepSeek, Qwen) may also pose accessibility challenges for some researchers, though these models are increasingly open-source.
Limitations
The pipeline relies on the quality of the underlying TTS and LLM models; if the TTS produces unnatural prosody, the "naturalistic" claim is weakened. The evaluation is primarily on the Moshi model; generalization to other full-duplex architectures is not tested. The "intent" labels are derived from the LLM's authoring, so any errors in the LLM's understanding of conversational acts will propagate to the labels. The paper does not extensively discuss the computational cost of the LLM-based authoring and validation loop for large-scale corpus generation.
Broader impact
This work addresses a critical bottleneck in training full-duplex dialogue systems: the lack of high-quality, intent-labeled data. By providing a scalable pipeline to generate such data, the paper enables the development of more natural and responsive conversational AI. The focus on distinguishing between different types of overlap (barge-in vs. backchannel) is particularly important for user experience, as incorrect handling of these events leads to frustrating interactions. The methodology could be extended to other modalities or languages with appropriate configuration overlays. The paper presents a novel pipeline for synthesizing intent-labeled, full-duplex conversational speech by decoupling LLM-based semantic authoring from acoustic realization via forced alignment. This approach effectively addresses the supervision gap in existing corpora by providing precise, intent-aware labels for turn-taking events, leading to significant improvements in the performance of full-duplex speech models like Moshi in managing conversational floor occupancy.
Author of this paper? Send a correction.