Neither Silence nor Overlap Is Failure: Intent-Conditioned Evaluation of Turn-Taking in Full-Duplex Spoken Dialogue Models
Abstract
Benchmarks for full-duplex spoken dialogue models score turn-taking with binary fixed-window rules that reward immediate response or silence by completeness of the prior turn. We argue that the appropriateness of a response offset, whether delayed silence or anticipatory overlap, is conditional on the speaker's latent intent, identifiable only from that speaker's behavior. We introduce TACT, a benchmark of 9,728 episodes and 73.2 hours from five dyadic corpora; each episode carries dialogue history, a per-speaker memory profile, and an annotator-derived posterior over six intent classes. Scoring replaces binary windows with a strictly proper threshold-weighted continuous ranked probability score whose weights are intent-conditioned timing kernels fitted to human floor-transfer-offset distributions, proving boundedness, consistency, and binary reduction. Across eleven systems the best model reaches 0.47 against a human topline of 0.86, is nearly invariant to speaker profiles, and TACT agrees with human judgments at Spearman 0.81 versus 0.46 for binary metrics.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces TACT, a novel intent-conditioned benchmark and scoring framework for full-duplex spoken dialogue models that replaces binary turn-taking metrics with a strictly proper, continuous ranked probability score. By conditioning evaluation on the speaker's latent intent and behavioral memory, the authors demonstrate that current state-of-the-art models exhibit significant rigidity and over-eagerness, failing to adapt to conversational context as humans do, thereby providing a more rigorous and human-aligned standard for assessing dialogue timing.
Methodology
The paper proposes a theoretically grounded shift from binary, fixed-window turn-taking metrics to a continuous, intent-conditioned scoring framework (TACT). The core methodological contribution is the use of a threshold-weighted Continuous Ranked Probability Score (twCRPS) where the weights are derived from intent-specific human floor-transfer-offset (FTO) distributions. This is a sophisticated approach that correctly identifies that "silence" and "overlap" are not inherently good or bad, but depend on the speaker's latent intent (e.g., rhetorical vs. answer-seeking). The authors provide rigorous proofs for boundedness, strict propriety, and the reduction of their metric to existing binary metrics in degenerate cases, which is a strong theoretical foundation. The integration of a "memory profile" for each speaker to condition the intent posterior is a novel and practical addition that addresses the variability in human conversational styles.
Results & evaluation
The experimental setup is robust, utilizing a large-scale benchmark (9,728 episodes, 73.2 hours) constructed from five diverse public corpora. The evaluation of eleven frontier systems (including open-source and proprietary APIs) provides a comprehensive landscape of current full-duplex model capabilities. The finding that models exhibit an "over-eagerness pathology" and fail to adapt to speaker-specific memory profiles is a significant empirical contribution. The validity study, showing a Spearman correlation of 0.81 with human judgments compared to 0.46 for binary metrics, strongly supports the utility of the proposed benchmark.
Reproducibility
The paper claims that data, kernels, and prompts are available in the Supplementary Material, which is a positive sign for reproducibility. However, the reliance on proprietary APIs (GPT-4o, Gemini) for some evaluations limits full reproducibility for those specific data points. The detailed description of the annotation protocol and the mathematical formulation of the scoring rules enhances the potential for independent verification.
Limitations
The primary limitation is the discretization of intent into six classes, which may oversimplify the continuous nature of human intent. The reliance on an LLM judge for fusing the intent posterior introduces potential biases or failure modes shared with the underlying LLM. Additionally, the kernels are fitted on English corpora, limiting the immediate generalizability to other languages without refitting. The "memory profile" mechanism, while effective in ablation, adds complexity to the inference pipeline for evaluated systems.
Broader impact
This work has high potential impact on the development of natural spoken dialogue systems. By providing a more nuanced and human-aligned evaluation metric, it can guide the training of models that are not just fast, but contextually appropriate. The identification of the "over-eagerness" problem highlights a critical gap in current model architectures that future research must address. The framework could be extended to other interactive AI domains where timing and intent are crucial. The paper introduces TACT, a novel intent-conditioned benchmark and scoring framework for full-duplex spoken dialogue models that replaces binary turn-taking metrics with a strictly proper, continuous ranked probability score. By conditioning evaluation on the speaker's latent intent and behavioral memory, the authors demonstrate that current state-of-the-art models exhibit significant rigidity and over-eagerness, failing to adapt to conversational context as humans do, thereby providing a more rigorous and human-aligned standard for assessing dialogue timing.
Author of this paper? Send a correction.