EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents
Abstract
Voice agents, artificial intelligence systems that conduct spoken conversations to complete tasks, are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses two core evaluation challenges: generating realistic simulated conversations, and measuring quality across the full scope of voice-specific failure modes. We present EVA-Bench, an end-to-end evaluation framework that addresses both. On the simulation side, EVA-Bench orchestrates bot-to-bot audio conversations over dynamic multi-turn dialogues, with automatic simulation validation that detects user simulator error and appropriately regenerates conversations before scoring. On the measurement side, EVA-Bench introduces two composite metrics: EVA-A (Accuracy), capturing task completion, faithfulness, and audio-level speech fidelity; and EVA-X (Experience), capturing conversation progression, spoken conciseness, and turn-taking timing. Both metrics apply to different agent architectures, enabling direct cross-architecture comparison. EVA-Bench includes 213 scenarios across three enterprise domains, a controlled perturbation suite for accent and noise robustness, and pass@1, pass@k, pass^k measurements that distinguish peak from reliable capability. Across 12 systems spanning all three architectures, we find: (1) no system simultaneously exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1; (2) peak and reliable performance diverge substantially (median pass@k - pass^k gap of 0.44 on EVA-A); and (3) accent and noise perturbations expose substantial robustness gaps, with effects varying across architectures, systems, and metrics (mean up to 0.314). We release the full framework, evaluation suite, and benchmark data under an open-source license.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of EVA-Bench, a novel evaluation framework for voice agents that combines realistic simulation with comprehensive metrics to assess performance across various architectures and conditions. This work significantly advances the field by addressing critical evaluation challenges and providing a foundation for future research in voice agent technology.
Methodology
The methodology presented in this paper is robust and addresses significant gaps in the evaluation of voice agents. The authors introduce an end-to-end framework, EVA-Bench, that combines realistic bot-to-bot audio simulation with comprehensive measurement metrics (EVA-A and EVA-X). The simulation methodology is particularly noteworthy as it incorporates automated validation to ensure the quality of user simulations, which is critical for obtaining reliable evaluation scores. The introduction of controlled perturbations to assess robustness against accent and noise variations further strengthens the methodology, allowing for a nuanced understanding of system performance across different conditions.
Results & evaluation
The experimental evaluation is thorough, involving 12 systems across three distinct architectures and a total of 213 scenarios. The results reveal critical insights into the performance of voice agents, particularly the divergence between peak and reliable performance, which is a crucial finding for real-world applications. The use of multiple trials and the introduction of pass@1, pass@k, and pass^k metrics provide a comprehensive view of system capabilities, although the paper could benefit from additional comparative analysis against existing benchmarks to contextualize the findings further.
Reproducibility
The authors emphasize reproducibility by providing open-source access to the framework, evaluation suite, and benchmark data. They include detailed implementation instructions and configurations, which are essential for other researchers to replicate the study. However, the reliance on commercial model APIs for full reproduction may limit accessibility for some researchers, potentially impacting the overall reproducibility of the findings.
Limitations
The paper acknowledges several limitations, including potential biases in the LLM-based judges, the lack of multilingual coverage, and the constraints of the user simulator in replicating real human caller behaviors. Additionally, the evaluation does not account for harmful outputs or sensitive information exposure, which is particularly relevant in high-stakes domains. The authors also note that the framework does not assess more complex agent configurations, which may limit its applicability in certain scenarios.
Broader impact
The EVA-Bench framework has significant implications for the development and evaluation of voice agents in enterprise applications. By providing a comprehensive evaluation methodology, it can help improve the reliability and user experience of voice agents, ultimately leading to better deployment in real-world settings. The findings regarding performance gaps and robustness under perturbations can inform future research and development efforts, guiding improvements in voice agent architectures and evaluation practices. The main contribution of this paper is the introduction of EVA-Bench, a novel evaluation framework for voice agents that combines realistic simulation with comprehensive metrics to assess performance across various architectures and conditions. This work significantly advances the field by addressing critical evaluation challenges and providing a foundation for future research in voice agent technology.
Author of this paper? Send a correction.