Back to Basics: Revisiting ASR in the Age of Voice Agents
Abstract
Automatic speech recognition (ASR) systems have achieved near-human accuracy on curated benchmarks, yet still fail in real-world voice agents under conditions that current evaluations do not systematically cover. Without diagnostic tools that isolate specific failure factors, practitioners cannot anticipate which conditions, in which languages, will cause what degree of degradation. We introduce WildASR, a multilingual (four-language) diagnostic benchmark sourced entirely from real human speech that factorizes ASR robustness along three axes: environmental degradation, demographic shift, and linguistic diversity. Evaluating seven widely used ASR systems, we find severe and uneven performance degradation, and model robustness does not transfer across languages or conditions. Critically, models often hallucinate plausible but unspoken content under partial or degraded inputs, creating concrete safety risks for downstream agent behavior. Our results demonstrate that targeted, factor-isolated evaluation is essential for understanding and improving ASR reliability in production systems. Besides the benchmark itself, we also present three analytical tools that practitioners can use to guide deployment decisions.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents WildASR, a comprehensive benchmark for evaluating ASR systems under realistic conditions, revealing critical shortcomings in current models and providing tools for practitioners to enhance system reliability. The methodology is well-structured, and the findings have substantial implications for the future of ASR technology in voice agents.
Methodology
The paper introduces a novel benchmark, WildASR, which systematically evaluates ASR systems under real-world conditions by isolating failure factors across environmental, demographic, and linguistic dimensions. The methodology is robust, relying on real human speech data and controlled perturbations to ensure authenticity and relevance. The analytical tools provided for practitioners enhance the practical applicability of the research.
Results & evaluation
The experiments involved evaluating seven ASR systems across multiple languages and conditions, revealing significant performance degradation in real-world scenarios. The results are comprehensive, showcasing the limitations of current ASR models and emphasizing the need for targeted improvements. The use of various metrics, including word error rates and hallucination error rates, strengthens the evaluation.
Reproducibility
The paper outlines a clear curation pipeline and evaluation protocol, which aids in reproducibility. However, the lack of specific institutional affiliations and the absence of a publicly accessible demo may hinder broader adoption and verification by the community.
Limitations
The benchmark currently covers only four languages and is limited in demographic representation, particularly for children and older adults. The authors acknowledge the need for broader language and condition coverage in future work.
Broader impact
This research has significant implications for the deployment of ASR systems in real-world applications, particularly in voice agents. By highlighting critical failure modes and providing tools for analysis, it sets a foundation for improving ASR reliability, which is crucial for user safety and experience in conversational AI. The paper presents WildASR, a comprehensive benchmark for evaluating ASR systems under realistic conditions, revealing critical shortcomings in current models and providing tools for practitioners to enhance system reliability. The methodology is well-structured, and the findings have substantial implications for the future of ASR technology in voice agents.
Author of this paper? Send a correction.