MURMUR: An Efficient Inference System for Long-Form ASR
Abstract
Long-form automatic speech recognition (ASR) requires both high accuracy and low latency, but existing systems force a trade-off between the two. Chunk-based pipelines process audio in parallel windows for low latency, but lose cross-chunk context and need brittle heuristics to align speakers and timestamps at boundaries. Long-context ASR models resolve everything in a single pass for better accuracy, but are an order of magnitude slower. We propose Murmur, an inference system that overcomes this trade-off by operating at two levels. At the inter-chunk level, we revisit the chunk-based pipeline for modern long-context ASR, treating chunk size as a tunable hyperparameter, and show that intermediate chunk sizes strike a good balance of accuracy and latency. At the intra-chunk level, we exploit attention sparsity through a sliding window KV cache eviction policy applied to both output and speech tokens. On AMI-IHM, Murmur matches single-pass accuracy while reducing latency by 4.2x, with further gains from token eviction at less than 1% relative tcpWER degradation. The code of Murmur is available at https://github.com/uw-syfi/Murmur.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents MURMUR, an innovative inference system for long-form ASR that effectively balances accuracy and latency through a dual-level optimization approach. This contribution is significant in advancing the state-of-the-art in speech recognition systems, particularly in real-world applications where both speed and precision are critical.
Methodology
The methodology presented in the paper is well-structured and innovative, focusing on a two-level approach to improve long-form ASR systems. The inter-chunk and intra-chunk strategies effectively balance latency and accuracy, leveraging tunable chunk sizes and attention sparsity through a sliding window KV cache eviction policy. This dual-level optimization is a significant advancement over traditional ASR systems, which typically rely on either chunk-based or single-pass approaches without such nuanced control over context management.
Results & evaluation
The experiments are comprehensive, utilizing multiple datasets (AMI-IHM, Tedlium3, Earnings21) to validate the proposed method. The results demonstrate a clear performance improvement in terms of latency reduction while maintaining competitive accuracy, particularly in challenging conditions. The evaluation metrics, especially tcpWER, are appropriate for assessing the system's performance in real-world applications. The paper also includes a thorough comparison with existing systems like WhisperX, highlighting the advantages of the proposed approach.
Reproducibility
The implementation details are well-documented, with a link to the code repository provided. The paper describes the experimental setup, including the model architecture and configurations used, which enhances reproducibility. However, the reliance on specific datasets and the absence of a broader range of languages may limit the generalizability of the findings.
Limitations
The paper acknowledges limitations, including the focus on English ASR and the challenges posed by different prosodic structures in other languages. Additionally, the method's reliance on specific long-context ASR models may restrict its applicability to a broader range of systems. The potential for catastrophic failures in single-pass models is also noted, which could impact deployment in real-world scenarios.
Broader impact
The proposed system has significant implications for various applications, including meeting transcription, lecture notes, and clinical documentation. By improving the efficiency and accuracy of long-form ASR, the work could enhance accessibility and usability in professional settings. The findings may also motivate further research into adaptive context management and the exploration of ASR in diverse languages and acoustic environments. The paper presents MURMUR, an innovative inference system for long-form ASR that effectively balances accuracy and latency through a dual-level optimization approach. This contribution is significant in advancing the state-of-the-art in speech recognition systems, particularly in real-world applications where both speed and precision are critical.
Author of this paper? Send a correction.