WearVox: An Egocentric Multichannel Voice Assistant Benchmark for Wearables
Abstract
Wearable devices such as AI glasses are transforming voice assistants into always-available, hands-free collaborators that integrate seamlessly with daily life, but they also introduce challenges like egocentric audio affected by motion and noise, rapid micro-interactions, and the need to distinguish device-directed speech from background conversations. Existing benchmarks largely overlook these complexities, focusing instead on clean or generic conversational audio. To bridge this gap, we present WearVox, the first benchmark designed to rigorously evaluate voice assistants in realistic wearable scenarios. WearVox comprises 3,842 multi-channel, egocentric audio recordings collected via AI glasses across five diverse tasks including Search-Grounded QA, Closed-Book QA, Side-Talk Rejection, Tool Calling, and Speech Translation, spanning a wide range of indoor and outdoor environments and acoustic conditions. Each recording is accompanied by rich metadata, enabling nuanced analysis of model performance under real-world constraints. We benchmark leading proprietary and open-source speech Large Language Models (SLLMs) and find that most real-time SLLMs achieve accuracies on WearVox ranging from 29% to 59%, with substantial performance degradation on noisy outdoor audio, underscoring the difficulty and realism of the benchmark. Additionally, we conduct a case study with two new SLLMs that perform inference with single-channel and multi-channel audio, demonstrating that multi-channel audio inputs significantly enhance model robustness to environmental noise and improve discrimination between device-directed and background speech. Our results highlight the critical importance of spatial audio cues for context-aware voice assistants and establish WearVox as a comprehensive testbed for advancing wearable voice AI research.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces WearVox, a pioneering benchmark for evaluating voice assistants in wearable contexts, significantly advancing the understanding of the challenges and requirements for robust voice interaction in real-world environments. The comprehensive dataset and rigorous evaluation methodology position this work as a critical resource for future research in wearable AI systems.
Methodology
The methodology presented in this paper is robust, with a clear focus on creating a comprehensive benchmark dataset (WearVox) that addresses the unique challenges of wearable voice assistants. The authors effectively detail the data collection process, including the use of egocentric multi-channel audio and the diverse set of tasks designed to simulate real-world scenarios. The inclusion of various speaker roles and acoustic conditions adds depth to the dataset, making it a valuable resource for future research. However, the paper could benefit from more explicit details on the data collection process and the specific algorithms used for model evaluations.
Results & evaluation
The experimental evaluation is thorough, benchmarking multiple state-of-the-art speech Large Language Models (SLLMs) against the WearVox dataset. The results demonstrate significant variability in model performance across different tasks and environments, highlighting the challenges posed by real-world conditions. The case study comparing single-channel and multi-channel models provides valuable insights into the advantages of multi-channel audio processing. However, the paper does not provide a detailed statistical analysis of the results, which would strengthen the findings.
Reproducibility
While the paper describes the methodology and experimental setup in detail, it lacks a clear path for reproducibility, such as providing access to the dataset or code. The absence of a project URL or demo further limits the ability of other researchers to replicate the study. Providing these resources would enhance the paper's impact and utility in the research community.
Limitations
One limitation is the reliance on a specific hardware setup (AI glasses) for data collection, which may not generalize to all wearable devices. Additionally, the dataset may not cover all possible real-world scenarios, and the performance of models on unseen data remains uncertain. The paper also does not address potential biases in the dataset, which could affect the generalizability of the results.
Broader impact
The development of the WearVox benchmark has significant implications for advancing the field of wearable AI and voice assistants. By addressing the unique challenges of egocentric audio processing, the research can lead to more robust and context-aware voice assistants that seamlessly integrate into users' daily lives. The findings may also influence future research directions, including the incorporation of multimodal signals and the exploration of new tasks for wearable assistants. This paper introduces WearVox, a pioneering benchmark for evaluating voice assistants in wearable contexts, significantly advancing the understanding of the challenges and requirements for robust voice interaction in real-world environments. The comprehensive dataset and rigorous evaluation methodology position this work as a critical resource for future research in wearable AI systems.
Author of this paper? Send a correction.