Voice Evaluation of Reasoning Ability: Diagnosing the Modality-Induced Performance Gap
Abstract
We present Voice Evaluation of Reasoning Ability (VERA), a benchmark for evaluating reasoning ability in voice-interactive systems under real-time conversational constraints. VERA comprises 2,931 voice-native episodes derived from established text benchmarks and organized into five tracks (Math, Web, Science, Long-Context, Factual). Each item is adapted for speech interaction while preserving reasoning difficulty. VERA enables direct text-voice comparison within model families and supports analysis of how architectural choices affect reliability. We assess 12 contemporary voice systems alongside strong text baselines and observe large, consistent modality gaps: on competition mathematics a leading text model attains 74.8% accuracy while its voice counterpart reaches 6.1%; macro-averaged across tracks the best text models achieve 54.0% versus 11.3% for voice. Latency-accuracy analyses reveal a low-latency plateau, where fast voice systems cluster around ~10% accuracy, while approaching text performance requires sacrificing real-time interaction. Diagnostic experiments indicate that common mitigations are insufficient. Increasing "thinking time" yields negligible gains; a decoupled cascade that separates reasoning from narration improves accuracy but still falls well short of text and introduces characteristic grounding/consistency errors. Failure analyses further show distinct error signatures across native streaming, end-to-end, and cascade designs. VERA provides a reproducible testbed and targeted diagnostics for architectures that decouple thinking from speaking, offering a principled way to measure progress toward real-time voice assistants that are both fluent and reliably reasoned.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a comprehensive evaluation of reasoning ability in voice-interactive systems, introducing the VERA benchmark and systematically documenting the Voice Reasoning Gap. The methodology and experimental rigor provide valuable insights into the architectural challenges faced by current voice models, paving the way for future innovations in the field.
Methodology
The methodology is robust and well-structured, introducing the Voice Evaluation of Reasoning Ability (VERA) benchmark, which is a significant advancement in evaluating reasoning capabilities of voice-interactive systems. The authors have meticulously adapted existing text benchmarks for voice interaction, ensuring that the reasoning difficulty is preserved while addressing the unique challenges posed by real-time conversational constraints. The multi-stage pipeline for adapting benchmarks is particularly noteworthy, as it emphasizes reproducibility and quality control in the dataset creation process. The systematic evaluation of various voice systems against text baselines provides a comprehensive framework for understanding the Voice Reasoning Gap (VRG).
Results & evaluation
The experimental setup is thorough, assessing 12 contemporary voice systems alongside strong text baselines, which allows for a direct comparison of performance across modalities. The results reveal significant modality gaps, with detailed analyses of latency-accuracy trade-offs and failure modes. The use of statistical validation methods, such as McNemar's test, adds rigor to the findings. The paper also includes a well-defined error taxonomy that categorizes failures, providing insights into the underlying architectural challenges faced by voice systems.
Reproducibility
The authors have made efforts to ensure reproducibility by providing a clear description of the benchmark construction process and making the dataset available on GitHub. However, the paper does not provide detailed implementation instructions for the models evaluated, which may hinder full reproducibility of the experimental results.
Limitations
One limitation is the focus on existing voice systems without exploring potential novel architectures that could bridge the VRG. Additionally, the paper acknowledges that the evaluation is not a controlled experiment designed to prove causality, which may limit the generalizability of the findings. The reliance on commercial systems may also introduce variability that is not fully accounted for in the analysis.
Broader impact
The findings have significant implications for the development of voice-interactive systems, particularly in enhancing reasoning capabilities in real-time applications. The VERA benchmark can serve as a critical tool for future research, guiding the design of more effective voice assistants that balance fluency and reasoning depth. The insights gained from this work could influence the direction of research in voice technology and artificial intelligence, potentially leading to more intelligent and conversational systems. This paper presents a comprehensive evaluation of reasoning ability in voice-interactive systems, introducing the VERA benchmark and systematically documenting the Voice Reasoning Gap. The methodology and experimental rigor provide valuable insights into the architectural challenges faced by current voice models, paving the way for future innovations in the field.
Author of this paper? Send a correction.