MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models
Abstract
Full-Duplex Speech Language Models (FD-SLMs) enable real-time, overlapping conversational interactions, offering a more dynamic user experience compared to traditional half-duplex models. However, existing benchmarks primarily focus on evaluating single-round interactions, neglecting the complexities of multi-round communication. Evaluating FD-SLMs in multi-round settings poses significant challenges, including blurred turn boundaries in communication and context inconsistency during model inference. Also, existing benchmarks often focus solely on evaluating conversational features, neglecting other critical aspects. To address these gaps, we introduce MTR-DuplexBench, a novel benchmark designed for a comprehensive multi-round evaluation of FD-SLMs. MTR-DuplexBench not only segments continuous full-duplex dialogues into discrete turns for turn-by-turn assessment but also incorporates various evaluation aspects, including conversational features, dialogue quality, instruction following, and safety. Experimental results reveal that current FD-SLMs face difficulties in maintaining consistent performance across multiple rounds and evaluation dimensions, highlighting the necessity and effectiveness of our benchmark. Code and data are available at: https://github.com/ZhangHe0918/MTR-DuplexBench
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of MTR-DuplexBench, a comprehensive benchmark for evaluating full-duplex speech language models in multi-round conversational settings, addressing critical gaps in existing evaluation methodologies. The technical contributions, innovative methodology, and thorough experimental validation position this work as a significant advancement in the field of audio and speech processing.
Methodology
The proposed methodology introduces a novel turn segmentation algorithm that effectively addresses the challenges of blurred turn boundaries and context inconsistency in full-duplex speech interactions. This is a significant advancement over existing benchmarks, which primarily focus on single-round evaluations. The comprehensive evaluation framework that incorporates multiple dimensions (conversational features, dialogue quality, instruction following, and safety) is a notable strength, providing a holistic view of FD-SLM performance.
Results & evaluation
The experimental evaluation is thorough, utilizing a variety of datasets and metrics to assess the performance of existing FD-SLMs against the new benchmark. The results highlight the limitations of current models in handling multi-round conversations, which underscores the relevance and necessity of the proposed benchmark. However, the reliance on synthetic data for some evaluations may limit the generalizability of findings.
Reproducibility
The paper provides a GitHub repository with code and data, which enhances reproducibility. The detailed descriptions of the evaluation metrics and experimental setups allow for other researchers to replicate the experiments. However, the paper could benefit from more explicit instructions on the environment setup and dependencies.
Limitations
The benchmark relies on a combination of natural and synthetic datasets, which may not fully capture the diversity of real-world conversations. Additionally, the focus on English speech limits applicability to multilingual contexts. The methodology also introduces a gap between guided and unguided evaluation trajectories due to the use of teacher forcing in some evaluations.
Broader impact
The proposed benchmark has the potential to significantly advance the field of full-duplex speech language models by providing a more comprehensive evaluation framework. This can lead to the development of more robust and capable FD-SLMs that can better handle real-world conversational dynamics, ultimately improving user experience in applications such as virtual assistants and interactive voice response systems. The main contribution of this paper is the introduction of MTR-DuplexBench, a comprehensive benchmark for evaluating full-duplex speech language models in multi-round conversational settings, addressing critical gaps in existing evaluation methodologies. The technical contributions, innovative methodology, and thorough experimental validation position this work as a significant advancement in the field of audio and speech processing.
Author of this paper? Send a correction.