VibeVoice-ASR-BitNet Technical Report
Abstract
We present VibeVoice-ASR-BitNet, a compressed variant of VibeVoice-ASR optimized for real-time inference on edge CPUs. We apply heterogeneous quantization tailored to the computational characteristics of each stage: the VAE acoustic tokenizer uses full-pipeline INT8 quantization (I8_S) with kernel fusion and SIMD optimization, while the autoregressive language model adopts BitNet-style ternary weights (I2_S). To preserve accuracy under aggressive compression, we employ a progressive quantization-aware training strategy. For inference, we implement custom SIMD kernels and fused operators within the ggml framework targeting both ARM and x86 platforms, achieving real-time recognition with RTF < 1 using as few as 3 CPU threads. VibeVoice-ASR-BitNet is 1.6-2.3x faster than Whisper.cpp at comparable model sizes (~1.6 GB), with only modest accuracy degradation compared to the FP16 baseline.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a highly effective engineering solution for deploying large LLM-based ASR models on edge CPUs through heterogeneous quantization (INT8 for VAE, ternary for LM) and custom SIMD optimization, achieving real-time performance with minimal accuracy loss and significant compression.
Methodology
The paper proposes a heterogeneous quantization strategy for the VibeVoice-ASR architecture, specifically targeting edge CPU deployment. The core technical contribution lies in the distinct treatment of the two main components: the VAE tokenizer is quantized to full-pipeline INT8 (I8_S) to address its activation-bound nature, while the autoregressive language model (LM) decoder utilizes BitNet-style ternary weights (I2_S) to address its weight-bound nature during decoding. The authors introduce a progressive quantization-aware training (QAT) schedule using a blending parameter to stabilize training, which is a practical engineering contribution. The inference engine is built on ggml with custom SIMD kernels (AVX2/NEON) for efficient INT8 and ternary (unpacking to INT8 for MAD operations) computation. The methodology is sound and well-reasoned, leveraging established techniques (BitNet, ggml, INT8 quantization) in a novel combination specific to the VAE-LM ASR paradigm.
Results & evaluation
The evaluation is conducted on a server-grade AMD EPYC CPU, which serves as a proxy for high-end edge hardware, though real-world edge devices (e.g., mobile SoCs) may have different memory bandwidth and cache characteristics. The paper reports a 2.9x compression ratio (4.62 GB to 1.58 GB) and achieves Real-Time Factor (RTF) < 1 with only 3 threads, demonstrating significant efficiency gains over FP16 baselines and competitive performance against Whisper.cpp. Accuracy is evaluated on multiple multilingual benchmarks (MLC, AISHELL, etc.), showing modest degradation (1-4%) compared to the larger FP16 baseline. The comparison with Whisper.cpp is relevant and highlights the benefits of the specific heterogeneous quantization approach for LLM-based ASR.
Reproducibility
The paper provides a GitHub link to the codebase (VibeASR.cpp) and Hugging Face links to the models. The methodology describes the quantization schemes, training schedules, and kernel optimizations in sufficient detail for replication by experts in the field. The use of standard frameworks (ggml) and well-defined quantization protocols enhances reproducibility.
Limitations
The primary limitation is the lack of streaming support, as noted by the authors. The current implementation is batch-oriented, which limits its applicability in real-time interactive scenarios where low latency is critical. Additionally, the evaluation is performed on a specific server CPU; performance on actual resource-constrained edge devices (ARM-based mobile chips, etc.) is not explicitly detailed, although ARM support is mentioned. The generalizability of the heterogeneous strategy to other VAE-LM architectures (e.g., Whisper) is also unexplored.
Broader impact
This work significantly advances the field of on-device AI by demonstrating that large, accurate LLM-based ASR systems can be deployed on commodity CPUs without GPUs. This has profound implications for privacy-preserving speech recognition, reduced cloud infrastructure costs, and enabling advanced voice assistants on low-power devices. It bridges the gap between high-accuracy LLM-based ASR and practical, efficient edge deployment. This paper presents a highly effective engineering solution for deploying large LLM-based ASR models on edge CPUs through heterogeneous quantization (INT8 for VAE, ternary for LM) and custom SIMD optimization, achieving real-time performance with minimal accuracy loss and significant compression.
Author of this paper? Send a correction.