VibeVoice-ASR-Streaming Technical Report
Abstract
Traditional speaker-attributed ASR systems treated ASR and speaker diarization as two separate tasks. Recently, end-to-end models such as VibeVoice-ASR have unified the two tasks within a single model. However, existing unified models still mainly support offline recognition, making it difficult to meet the low-latency requirements of real-time voice assistants and agents. To tackle this issue, we present VibeVoice-ASR-Streaming, one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. It interleaves fixed-size audio chunks, a small amount of lookahead audio and previous text. This allows the model to produce ''who said what'' as speech arrives, without a separate diarization stage. For transcription accuracy, our 7B model achieves the lowest average WER/CER across five evaluation sets. For speaker attribution, it achieves the best or tied-best on 12 of 13 evaluation settings. We release the 1.5B and 7B model weights together with inference code.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
VibeVoice-ASR-Streaming introduces a pioneering LLM-based end-to-end approach to streaming speaker-attributed ASR, achieving state-of-the-art performance in both transcription accuracy and speaker attribution with significantly lower latency than existing cloud services. The paper demonstrates that retaining full conversation history in the LLM context is sufficient for consistent speaker labeling, eliminating the need for separate diarization stages, and provides a rigorous analysis of the trade-offs between chunk size, lookahead, and model scale in streaming speech recognition.
Methodology
The paper proposes VibeVoice-ASR-Streaming, an end-to-end LLM-based framework for streaming speaker-attributed ASR. The core methodological contribution is the adaptation of the VibeVoice-ASR architecture (which unifies ASR and diarization) to a streaming setting by interleaving fixed-size audio chunks with generated text and retaining full conversation history in the LLM context. The authors introduce a fixed lookahead mechanism (4 latent frames) to handle boundary effects. The training strategy involves a three-stage process: non-streaming pre-training, streaming pre-training with chunked data, and streaming fine-tuning. The approach relies on the Qwen2.5 LLM backbone and dual tokenizers (Acoustic and Semantic) from the VibeVoice series. The methodology is sound, leveraging the strong contextual capabilities of LLMs to maintain speaker identity over long durations without explicit diarization modules, which is a significant architectural simplification compared to cascaded systems.
Results & evaluation
The experimental evaluation is comprehensive, covering four meeting benchmarks (AISHELL-4, AliMeeting, AMI-IHM, AMI-SDM) and the multilingual MLC-Challenge benchmark. The 7B model achieves the lowest average WER/CER among compared streaming systems and the best or tied-best cpWER/cpCER in 12 of 13 settings. The paper provides detailed ablations on chunk size, model scale, speaker-label placement, and lookahead depth. Notably, it compares against both specialized streaming ASR models and cloud services (Azure, Google), demonstrating a significant latency advantage (2.00s vs 8-9s) while maintaining or improving accuracy. The analysis of the "cost of streaming conversion" (degradation from offline to streaming) is particularly insightful, quantifying the trade-off.
Reproducibility
The paper is highly reproducible. The authors release both 1.5B and 7B model weights along with inference code. Detailed training data construction methods, including the use of Qwen3-ForcedAligner for word-level timing and specific augmentation strategies, are described. The exact hyperparameters for training stages are provided in the appendix. The evaluation protocol, including the specific versions of cloud services and normalization rules, is clearly defined.
Limitations
The primary limitations are the 8-minute recording length cap due to context window/compute constraints, the limited multilingual coverage (10 languages) dictated by the forced aligner, and performance degradation on long-duration overlaps due to the serialized output format. The first-packet latency (3.5s for 22-frame chunks) is higher than the steady-state latency, which may be a concern for real-time applications.
Broader impact
This work has significant impact on the development of real-time voice assistants and agents. By unifying ASR and speaker diarization in a streaming LLM, it removes the need for complex cascaded pipelines, simplifying system architecture and reducing latency. The open-sourcing of the models and code will likely accelerate research in streaming speech processing and enable the deployment of more robust, multi-speaker aware voice interfaces. VibeVoice-ASR-Streaming introduces a pioneering LLM-based end-to-end approach to streaming speaker-attributed ASR, achieving state-of-the-art performance in both transcription accuracy and speaker attribution with significantly lower latency than existing cloud services. The paper demonstrates that retaining full conversation history in the LLM context is sufficient for consistent speaker labeling, eliminating the need for separate diarization stages, and provides a rigorous analysis of the trade-offs between chunk size, lookahead, and model scale in streaming speech recognition.
Author of this paper? Send a correction.