Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play
Abstract
Role-play has become a key testbed for generative models, expanding from text-only dialogue to multimodal interaction. Extending role-play to speech captures prosody, emotion, and delivery, but also poses new evaluation challenges. Current pipelines often use audio large language models (ALLMs) as zero-shot judges, which miss paralinguistic cues, collapse multiple aspects into coarse scores, and rely on synthetic speech references that fail to reflect real-world roles. We present Speech-DRAME, a unified framework that contributes at three levels: (i) Speech-DRAME-EvalBench, an evaluation benchmark with bilingual human-annotated data and protocols for training and testing speech evaluation models (SEMs), (ii) DRAME-Eval, a fine-tuned evaluation model, which substantially outperforms zero-shot and few-shot ALLMs, and (iii) Speech-DRAME-RoleBench, a speech role-play benchmark that leverages DRAME-Eval as an automatic judge to compare speech foundation models (SFMs). Speech-DRAME distinguishes between two complementary evaluation strategies: Archetype Evaluation, a top-down approach measuring adherence to broad role archetypes, and Realism Evaluation, a bottom-up approach grounded in real human speech that emphasizes nuanced role quality. Compared to zero-shot ALLM judges, DRAME-Eval achieves stronger agreement with human ratings (Pearson correlation from 0.480 to 0.629 in archetypes, and 0.390 to 0.625 in realism). By integrating transparent benchmark resources, modeling approaches, and system-level evaluation, Speech-DRAME provides the first comprehensive, reproducible foundation for assessing spoken role-play.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of Speech-DRAME, a comprehensive framework for evaluating speech role-play that combines dual evaluation strategies and human-annotated datasets, significantly improving the assessment of generative models in speech interactions. This work addresses critical gaps in existing evaluation methodologies, paving the way for more effective and realistic speech-based AI systems.
Methodology
The methodology presented in the paper is robust and innovative, introducing a dual evaluation paradigm that combines archetype and realism assessments for speech role-play. The creation of DRAME-EvalBench and DRAME-Eval demonstrates a thoughtful approach to addressing the limitations of existing evaluation models by incorporating human-annotated data and fine-tuning techniques. The authors effectively leverage a comprehensive dataset that includes both synthetic and real human speech, which enhances the evaluation's relevance and applicability. The clear definitions and structured evaluation strategies are commendable, making the framework both systematic and reproducible.
Results & evaluation
The experimental evaluation is thorough, with a well-defined setup that includes zero-shot, few-shot, and fine-tuning conditions. The results demonstrate significant improvements in correlation with human ratings when using the DRAME-Eval model compared to existing ALLMs. The paper provides detailed statistical analyses and comparisons across various models, showcasing the effectiveness of the proposed methods. However, the reliance on proprietary models for some evaluations may limit the generalizability of the findings.
Reproducibility
The authors emphasize reproducibility by providing access to datasets, models, and code through a GitHub repository. The detailed descriptions of the experimental setup, including data collection, annotation protocols, and evaluation metrics, enhance the likelihood that other researchers can replicate the study. The commitment to transparency is evident, although the use of proprietary models may pose challenges for full reproducibility in certain aspects.
Limitations
One notable limitation is the potential bias introduced by the reliance on proprietary models for evaluation, which may not be accessible to all researchers. Additionally, while the dual evaluation framework is innovative, the complexity of the evaluation process may pose challenges for broader adoption in the community. The paper also acknowledges the gap between synthetic and real human performance, indicating that further work is needed to bridge this divide.
Broader impact
The proposed framework has significant implications for the development and evaluation of speech-based generative models, particularly in applications such as education, entertainment, and human-AI interaction. By providing a comprehensive and nuanced approach to evaluating speech role-play, the work encourages the creation of more sophisticated and human-aligned AI systems. The integration of human annotations and the focus on realistic speech delivery can lead to advancements in the quality and reliability of conversational agents. The main contribution of this paper is the introduction of Speech-DRAME, a comprehensive framework for evaluating speech role-play that combines dual evaluation strategies and human-annotated datasets, significantly improving the assessment of generative models in speech interactions. This work addresses critical gaps in existing evaluation methodologies, paving the way for more effective and realistic speech-based AI systems.
Author of this paper? Send a correction.