WESR: Scaling and Evaluating Word-level Event-Speech Recognition
Abstract
Speech conveys not only linguistic information but also rich non-verbal vocal events such as laughing and crying. While semantic transcription is well-studied, the precise localization of non-verbal events remains a critical yet under-explored challenge. Current methods suffer from insufficient task definitions with limited category coverage and ambiguous temporal granularity. They also lack standardized evaluation frameworks, hindering the development of downstream applications. To bridge this gap, we first develop a refined taxonomy of 21 vocal events, with a new categorization into discrete (standalone) versus continuous (mixed with speech) types. Based on the refined taxonomy, we introduce WESR-Bench, an expert-annotated evaluation set (900+ utterances) with a novel position-aware protocol that disentangles ASR errors from event detection, enabling precise localization measurement for both discrete and continuous events. We also build a strong baseline by constructing a 1,700+ hour corpus, and train specialized models, surpassing both open-source audio-language models and commercial APIs while preserving ASR quality. We anticipate that WESR will serve as a foundational resource for future research in modeling rich, real-world auditory scenes.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a comprehensive framework for Word-level Event-Speech Recognition, contributing significantly to the understanding of non-verbal vocal events in speech processing. The methodology is innovative, and the experimental results demonstrate a strong technical impact, making it a valuable resource for future research in the field.
Methodology
The paper introduces a novel framework for Word-level Event-Speech Recognition (WESR), which includes a refined taxonomy of 21 vocal events and a new evaluation set called WESR-Bench. The methodology is well-structured, focusing on both the modeling aspect and the evaluation protocol. The authors successfully disentangle ASR errors from event detection, allowing for precise localization of both discrete and continuous vocal events. The hybrid retrieval strategy for dataset construction and the rigorous human annotation process enhance the credibility of the proposed methods.
Results & evaluation
The experiments are comprehensive, with a large-scale dataset (WESR-Train) and a robust evaluation framework (WESR-Bench). The authors demonstrate that their models outperform existing open-source and commercial APIs while maintaining ASR quality. The results are presented clearly, with detailed comparisons across various models, showcasing the effectiveness of their approach in detecting non-verbal vocal events.
Reproducibility
The paper provides sufficient implementation details, including training parameters and data curation strategies, which contribute to reproducibility. However, the reliance on commercial APIs for data quality may limit accessibility for some researchers. The authors emphasize ethical considerations and provide a GitHub repository for further exploration, which is a positive aspect for reproducibility.
Limitations
The paper acknowledges limitations related to language coverage, as the system is currently verified only for English and Chinese. Additionally, the resource intensity of the dataset construction may pose challenges for scalability and reproducibility in resource-constrained environments. The potential inclusion of inappropriate content in the dataset due to the large-scale annotation process is also a concern.
Broader impact
The proposed WESR framework has significant implications for applications in natural language processing, human-computer interaction, and multimedia content analysis. By enabling more nuanced understanding of speech through the integration of non-verbal vocal events, it could enhance the development of more sophisticated conversational agents and accessibility tools. The work lays a foundation for future research in modeling rich auditory scenes, potentially influencing various fields such as linguistics, psychology, and AI ethics. The paper presents a comprehensive framework for Word-level Event-Speech Recognition, contributing significantly to the understanding of non-verbal vocal events in speech processing. The methodology is innovative, and the experimental results demonstrate a strong technical impact, making it a valuable resource for future research in the field.
Author of this paper? Send a correction.