ParaASR: Multi-Token Prediction for Fast and Long-Context LLM-Based Speech Recognition
Abstract
Audio-encoder-LLM-decoder architectures have become the dominant paradigm for modern automatic speech recognition (ASR), improving transcription quality through large-scale language modeling. However, the cost of autoregressive decoding scales with decoder size, creating a fundamental trade-off between recognition quality and serving latency. We argue this trade-off is not inherent: unlike open-ended text generation, ASR outputs are strongly anchored to the input speech signal, providing a natural inductive bias toward high-parallelism decoding. Building on this, we introduce ParaASR, an ASR system that leverages Multi-Token Prediction (MTP) to let a 4B LLM decoder emit multiple tokens per forward step. Starting from a publicly available audio-language foundation, the model first establishes a robust autoregressive recognizer and then aligns five future-token branches through a staged optimization recipe. At inference, it proposes a six-token continuation per step and admits only the verified prefix into the transcript, preserving the safety of standard autoregressive decoding. The average accepted length reaches 5.0 out of 6 proposed tokens, confirming that the deterministic structure of speech makes ASR an especially natural setting for multi-token decoding. ParaASR further retains a native 32K-context window and transcribes up to 30 minutes of audio in a single pass. Across diverse benchmarks, it attains average error rates of 2.97%, 3.68%, and 3.70% on Chinese, English, and long-form evaluations, respectively, while reaching a real-time factor (RTF) as low as 0.0053. These results show that decoder scaling, low-latency inference, and long-context transcription need not be competing goals when future-token proposals are anchored by the acoustic signal and guarded by autoregressive verification.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
ParaASR effectively adapts multi-token prediction to ASR, achieving a compelling balance between state-of-the-art accuracy and low-latency inference, thereby challenging the perceived trade-off between model scale and serving efficiency in LLM-based speech recognition.
Methodology
The paper proposes ParaASR, an LLM-based Automatic Speech Recognition (ASR) system that integrates Multi-Token Prediction (MTP) to accelerate inference. The core methodological contribution is the application of speculative decoding principles (specifically MTP) to the ASR domain, leveraging the deterministic nature of speech-to-text mapping to propose multiple future tokens per forward pass. The architecture consists of a frozen audio encoder, a linear adapter, and a 4B LLM decoder with five auxiliary MTP branches. The training recipe is staged: audio-language pretraining, ASR supervised fine-tuning (SFT), and finally MTP alignment. The MTP training involves a frozen-branch alignment stage followed by joint calibration. The approach is technically sound and logically motivated; the insight that ASR outputs are more predictable than open-ended text generation is valid and well-supported by the high acceptance rates reported. However, the specific architectural modifications (adding MTP heads) are incremental adaptations of existing speculative decoding techniques rather than a fundamentally new algorithmic breakthrough.
Results & evaluation
The evaluation covers a wide range of benchmarks including Chinese (AISHELL-1, AISHELL-2, WenetSpeech), English (LibriSpeech, Common Voice, VoxPopuli, Earnings22), and long-form ASR tasks. The results demonstrate state-of-the-art accuracy for an LLM-based ASR system, with low CER/WER scores. Crucially, the paper reports a Real-Time Factor (RTF) of 0.0053 on a single H800 GPU, which is a significant efficiency improvement over standard autoregressive decoding. The ablation studies confirm that MTP does not degrade accuracy and that the staged training is effective. The comparison against baselines like VibeVoice-ASR, FunASR-Nano, and Qwen3-ASR-1.7B is appropriate. The long-form evaluation using a constructed Wenet testnet set adds value, though the construction method (concatenating segments) is a simplification of true long-form challenges.
Reproducibility
The paper provides detailed descriptions of the training stages, data mixture (100K hours short-form, 50K hours long-form pseudo-labeled), and hyperparameters (learning rates, batch sizes). However, the use of proprietary datasets and the "publicly available audio-language foundation" (likely StepAudio, which is not fully open-weight or documented in sufficient detail for exact reproduction) limits reproducibility. The specific details of the long-form data curation pipeline (VAD, ROVER, LLM refinement) are described but the code for this pipeline is not provided. The model weights are not mentioned as open-sourced.
Limitations
The paper does not extensively discuss the computational overhead of the MTP branches during training or the memory footprint. The reliance on pseudo-labeled data for long-form training introduces potential error propagation, although the filtering mechanism ($e > 0.05$) mitigates this. The performance on code-switching and highly noisy environments is mentioned but specific metrics are not detailed in the abstract/text provided. The generalization of the high acceptance rate (5.0/6) to out-of-distribution domains (e.g., heavy accents, different languages not seen in training) is not fully explored.
Broader impact
This work demonstrates that large language models can be efficiently deployed for real-time speech recognition, potentially lowering the barrier for high-quality ASR services. It highlights the synergy between acoustic grounding and language modeling efficiency. The ability to transcribe long audio in a single pass has significant implications for meeting transcription, broadcast monitoring, and archival processing. ParaASR effectively adapts multi-token prediction to ASR, achieving a compelling balance between state-of-the-art accuracy and low-latency inference, thereby challenging the perceived trade-off between model scale and serving efficiency in LLM-based speech recognition.
Author of this paper? Send a correction.