Conversation Is a Two-Body Problem: Dyadic Evaluation of Full-Duplex Dialogue Models
Abstract
Full-duplex spoken dialogue models listen and speak at the same time, enabling voice agents to have natural, low-latency interactions that turn-based systems cannot offer. However, they are commonly evaluated against single-sided interlocutors: pre-recorded audio that cannot react, or an automated examiner that reacts in real time but only administers a fixed sequence of tests and is never graded. These single-sided frameworks evaluate only half of a two-body problem, where turn-taking, overlap, and interruption are joint products of two coupled speakers. We propose DyaFDB, a framework that evaluates full-duplex models in a dyadic setup: two models converse directly under assigned roles with cooperative or conflicting goals, and both sides are scored offline with an external judge. DyaFDB probes how the two models behave toward each other, such as how they take turns or carry an assigned role under different interests. We instantiate four tasks as 140 scenarios and record 7,560 conversations, covering six self- and cross-play pairings. Throughout the experiments, we observe that how a model behaves continually reshapes its partner. We thus demonstrate that each model must be both the examiner and examinee of the other, and no single fixed interlocutor can play both parts. We will release the scenarios, role prompts, and recording protocols between two full-duplex models, without any pre-recorded audio.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a dyadic evaluation framework for full-duplex dialogue models, demonstrating that conversational behaviors are joint properties of interacting agents rather than individual model traits, and providing a rigorous, reproducible benchmark for assessing these dynamics.
Methodology
The paper proposes DyaFDB, a dyadic evaluation framework for full-duplex spoken dialogue models. The core methodological innovation is the "two-body" approach: instead of evaluating a model against a static or scripted interlocutor, it pairs two live, free-running full-duplex models in a lockstep, frame-level bridge. This allows for the measurement of emergent conversational dynamics like turn-taking, interruption, and role adherence that are joint products of both speakers. The framework includes four distinct task families (Coordination and Conflict) with specific archetypes (e.g., information pooling, contested channel, secret keeping). The use of a lockstep clock (80ms frames) to synchronize two asynchronous models is a significant technical contribution, ensuring deterministic and reproducible interactions without network jitter. The evaluation protocol involves offline scoring by an LLM judge (Claude Opus-5) and mechanical metrics, with rigorous counterbalancing of roles and speaking order.
Results & evaluation
The experiments are extensive, involving 7,560 conversations (126 hours) across three open-weight models (PersonaPlex, MiniCPM-o, Raon-SpeechChat) in self-play and cross-play configurations. The results provide deep insights into model behavior, such as how a model's performance is heavily influenced by its partner (e.g., one model becomes a better defender against a specific partner but worse against another). The paper includes variance decomposition to quantify how much of the performance variance is due to the model itself versus the partner or interaction. It also compares speech model performance against text LLM backbones, highlighting the specific challenges of maintaining coherence and goal-directedness in the full-duplex audio modality. The inclusion of human expert validation for the LLM judge adds credibility to the automated scoring.
Reproducibility
The paper commits to releasing scenarios, role prompts, and recording protocols. It provides detailed implementation details for the bridge, adapters for different model interfaces, and the scoring pipeline. The use of a lockstep clock and specific VAD thresholds enhances reproducibility. However, the reliance on specific open-weight models that may change over time and the complexity of setting up the real-time audio bridge between different model architectures could pose challenges for external reproduction.
Limitations
The evaluation is limited to three specific open-weight models, which may not represent the full spectrum of full-duplex capabilities, especially compared to proprietary systems. The LLM judge, while validated, is still an automated proxy for human judgment and may miss subtle nuances in voice or prosody that affect perceived quality. The "lockstep" nature of the bridge, while good for control, might not perfectly replicate the jitter and latency variations of a real-world networked audio conversation.
Broader impact
This work shifts the paradigm for evaluating conversational AI from single-agent testing to interactive, multi-agent evaluation. It highlights that "full-duplex" capabilities cannot be assessed in isolation and provides a toolkit for the community to test robustness, social intelligence, and security (e.g., secret keeping) of voice agents. This is crucial as voice agents are deployed in increasingly sensitive and interactive contexts. The paper introduces a dyadic evaluation framework for full-duplex dialogue models, demonstrating that conversational behaviors are joint properties of interacting agents rather than individual model traits, and providing a rigorous, reproducible benchmark for assessing these dynamics.
Author of this paper? Send a correction.