NLE: Non-autoregressive LLM-based ASR by Transcript Editing
Abstract
While autoregressive (AR) LLM-based ASR systems achieve strong accuracy, their sequential decoding limits parallelism and incurs high latency. We propose NLE, a non-autoregressive (NAR) approach that formulates speech recognition as conditional transcript editing, enabling fully parallel prediction. NLE extracts acoustic embeddings and an initial hypothesis from a pretrained speech encoder, then refines the hypothesis using a bidirectional LLM editor trained with a latent alignment objective. An interleaved padding strategy exploits the identity mapping bias of Transformers, allowing the model to focus on corrections rather than full reconstruction. On the Open ASR leaderboard, NLE++ achieves 5.67% average WER with an RTFx (inverse real-time factor) of 1630. In single-utterance scenarios, NLE achieves 27x speedup over the AR baseline, making it suitable for real-time applications.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of a non-autoregressive LLM-based ASR system that effectively combines the strengths of pretrained speech encoders and language models through a novel editing approach, significantly improving transcription speed and maintaining competitive accuracy. The methodology is innovative, and the experimental results demonstrate substantial technical impact, making it a valuable contribution to the field of machine learning and speech recognition.
Methodology
The proposed methodology introduces a non-autoregressive (NAR) approach to automatic speech recognition (ASR) by framing it as conditional transcript editing. This is achieved through a bidirectional LLM editor that refines an initial hypothesis generated by a pretrained speech encoder. The interleaved padding strategy is a notable innovation, allowing the model to focus on corrections rather than full reconstructions, which enhances the efficiency of the editing process. The use of lightweight LoRA adapters for model adaptation is also a significant methodological contribution, enabling the model to leverage pretrained linguistic knowledge effectively while maintaining a manageable number of trainable parameters.
Results & evaluation
The experiments conducted are rigorous, with the authors evaluating their model against leading ASR systems on the Open ASR leaderboard. The reported results demonstrate a competitive word error rate (WER) of 5.67% for NLE++, with a substantial speedup of 27x over autoregressive baselines in single-utterance scenarios. The inclusion of ablation studies further strengthens the evaluation, providing insights into the impact of various design choices on performance. However, the paper could benefit from more extensive comparisons with a broader range of models and additional datasets to validate the robustness of the findings.
Reproducibility
The paper provides a detailed description of the model architecture, training procedures, and evaluation metrics, which enhances reproducibility. However, the lack of a publicly available code repository or demo URL limits the ability for others to directly replicate the results. The authors mention using specific datasets and configurations, which is helpful, but sharing the implementation would significantly improve reproducibility.
Limitations
The paper acknowledges that the NLE approach is less flexible than autoregressive models in scenarios requiring substantial changes to the hypothesis. It also highlights potential latency overhead due to the need for retokenization when using different tokenizers for the CTC encoder and the LLM. Moreover, the performance in multilingual settings appears to be weaker, suggesting that the model's training data may not be adequately representative of all languages.
Broader impact
The proposed NLE system has significant implications for real-time ASR applications, particularly in conversational settings where low latency is critical. By enabling faster and more accurate transcription, this approach could enhance user experiences in various domains, including virtual assistants, customer service, and accessibility technologies. The ability to refine initial hypotheses rather than regenerate them from scratch could also lead to more efficient use of computational resources. The main contribution of this paper is the introduction of a non-autoregressive LLM-based ASR system that effectively combines the strengths of pretrained speech encoders and language models through a novel editing approach, significantly improving transcription speed and maintaining competitive accuracy. The methodology is innovative, and the experimental results demonstrate substantial technical impact, making it a valuable contribution to the field of machine learning and speech recognition.
Author of this paper? Send a correction.