dLLM-ASR: A Faster Diffusion LLM-based Framework for Speech Recognition
Abstract
Automatic speech recognition (ASR) systems based on large language models (LLMs) achieve superior performance by leveraging pretrained LLMs as decoders, but their token-by-token generation mechanism leads to inference latency that grows linearly with sequence length. Meanwhile, discrete diffusion large language models (dLLMs) offer a promising alternative, enabling high-quality parallel sequence generation with pretrained decoders. However, directly applying native text-oriented dLLMs to ASR leads to a fundamental mismatch between open-ended text generation and the acoustically conditioned transcription paradigm required by ASR. As a result, it introduces unnecessary difficulty and computational redundancy, such as denoising from pure noise, inflexible generation lengths, and fixed denoising steps. We propose dLLM-ASR, an efficient dLLM-based ASR framework that formulates dLLM's decoding as a prior-guided and adaptive denoising process. It leverages an ASR prior to initialize the denoising process and provide an anchor for sequence length. Building upon this prior, length-adaptive pruning dynamically removes redundant tokens, while confidence-based denoising allows converged tokens to exit the denoising loop early, enabling token-level adaptive computation. Experiments demonstrate that dLLM-ASR achieves recognition accuracy comparable to autoregressive LLM-based ASR systems and delivers a 4.44$\times$ inference speedup, establishing a practical and efficient paradigm for ASR.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents dLLM-ASR, a novel framework that enhances speech recognition efficiency by integrating discrete diffusion large language models with adaptive denoising strategies, achieving a significant speedup while maintaining high accuracy. This work represents a meaningful contribution to the field of automatic speech recognition, addressing critical challenges and paving the way for future innovations in real-time applications.
Methodology
The proposed dLLM-ASR framework innovatively reformulates the ASR process by leveraging discrete diffusion large language models (dLLMs) in a way that addresses the inherent challenges of applying text-oriented models to speech recognition. The introduction of a prior-guided and adaptive denoising process is a significant methodological advancement, allowing for efficient token-level computation and dynamic sequence length adjustment. The confidence-based denoising and length-adaptive pruning strategies are particularly noteworthy, as they effectively reduce computational overhead while maintaining accuracy.
Results & evaluation
The experiments are comprehensive, utilizing multiple well-known datasets (LibriSpeech, CommonVoice, GigaSpeech) and benchmarks to evaluate the performance of dLLM-ASR against various baselines. The results demonstrate that dLLM-ASR achieves competitive recognition accuracy while significantly improving inference speed, showcasing a 4.44x speedup compared to autoregressive models. The ablation studies further validate the contributions of individual components of the framework.
Reproducibility
The paper provides detailed implementation specifics, including model architecture, training strategies, and hyperparameters, which facilitate reproducibility. However, the absence of a public code repository or demo limits the ease of replication by other researchers.
Limitations
While the paper presents a strong framework, it does not address potential limitations in terms of the model's performance on diverse languages or accents, which could affect generalizability. Additionally, the reliance on a specific pretrained dLLM may limit adaptability to other contexts or domains.
Broader impact
The advancements made in dLLM-ASR have significant implications for real-time speech recognition applications, particularly in scenarios where low latency is critical, such as virtual assistants, transcription services, and accessibility technologies. The methodology could inspire further research into integrating diffusion models in other areas of machine learning. The paper presents dLLM-ASR, a novel framework that enhances speech recognition efficiency by integrating discrete diffusion large language models with adaptive denoising strategies, achieving a significant speedup while maintaining high accuracy. This work represents a meaningful contribution to the field of automatic speech recognition, addressing critical challenges and paving the way for future innovations in real-time applications.
Author of this paper? Send a correction.