Sink or SWIM: Tackling Real-Time ASR at Scale
Abstract
Real-time automatic speech recognition systems are increasingly integrated into interactive applications, from voice assistants to live transcription services. However, scaling these systems to support multiple concurrent clients while maintaining low latency and high accuracy remains a major challenge. In this work, we present SWIM, a novel real-time ASR system built on top of OpenAI's Whisper model that enables true model-level parallelization for scalable, multilingual transcription. SWIM supports multiple concurrent audio streams without modifying the underlying model. It introduces a buffer merging strategy that maintains transcription fidelity while ensuring efficient resource usage. We evaluate SWIM in multi-client settings -- scaling up to 20 concurrent users -- and show that it delivers accurate real-time transcriptions in English, Italian, and Spanish, while maintaining low latency and high throughput. While Whisper-Streaming achieves a word error rate of approximately 8.2% with an average delay of approximately 3.4 s in a single-client, English-only setting, SWIM extends this capability to multilingual, multi-client environments. It maintains comparable accuracy with significantly lower delay -- around 2.4 s with 5 clients -- and continues to scale effectively up to 20 concurrent clients without degrading transcription quality and increasing overall throughput. Our approach advances scalable ASR by improving robustness and efficiency in dynamic, multi-user environments.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the development of SWIM, a scalable real-time ASR system that effectively manages multiple concurrent audio streams while maintaining low latency and high accuracy, thereby addressing a critical challenge in the field. The technical contributions and methodology are well-articulated, showcasing significant advancements in ASR technology.
Methodology
The paper introduces SWIM, a novel architecture that builds upon OpenAI's Whisper model, focusing on model-level parallelization to handle multiple concurrent audio streams. The methodology includes a buffer merging strategy that optimizes resource usage while preserving transcription fidelity. This approach is innovative in the context of real-time ASR systems, especially for multilingual applications, as it does not require modifications to the underlying model. The design choices are well-justified and align with the stated goals of scalability and efficiency.
Results & evaluation
The evaluation of SWIM is robust, demonstrating its capabilities in multi-client settings with up to 20 concurrent users. The reported results indicate a significant reduction in latency (from 3.4 seconds to 2.4 seconds with 5 clients) while maintaining comparable accuracy to existing systems. The experiments are well-structured, providing a clear comparison with Whisper-Streaming and highlighting the advantages of SWIM in practical scenarios.
Reproducibility
The paper lacks detailed implementation specifics, such as code availability or access to datasets used for testing. This omission raises concerns about the reproducibility of the results. While the methodology is sound, the absence of a public demo or project URL limits the ability for other researchers to validate the findings independently.
Limitations
One limitation noted is the reliance on the Whisper model, which may not generalize to all ASR tasks or languages. Additionally, while the system scales well, the paper does not address potential challenges in environments with highly variable audio quality or background noise, which could impact transcription accuracy.
Broader impact
The advancements presented in this paper have significant implications for real-time ASR applications, particularly in multilingual settings. By enabling efficient handling of multiple audio streams, SWIM could enhance user experiences in voice assistants, live transcription services, and other interactive applications. The work contributes to the ongoing evolution of scalable ASR systems, making them more accessible and effective for diverse user needs. The main contribution of this paper is the development of SWIM, a scalable real-time ASR system that effectively manages multiple concurrent audio streams while maintaining low latency and high accuracy, thereby addressing a critical challenge in the field. The technical contributions and methodology are well-articulated, showcasing significant advancements in ASR technology.
Author of this paper? Send a correction.