Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model
Abstract
Automatic speech recognition is dominated by autoregressive decoders that emit one token at a time. We ask whether a discrete diffusion language model can transcribe speech instead, refining a whole transcript in parallel over a small number of denoising steps. We train an audio-native interface for DiffusionGemma, a 26B mixture-of-experts model that generates text by uniform, random-token discrete diffusion rather than the absorbing-mask scheme common to recent diffusion language models. A frozen Whisper encoder supplies acoustic features, a lightweight projector maps them into the model embedding space, and low-rank adapters let the frozen backbone attend to the new modality. About 42M parameters are trained, which is 0.16 percent of the backbone. We find that the natural training objectives fail to ground the audio because their gradient reaches the projector only through attention that has already dismissed it. A connectionist temporal classification loss applied through the frozen output head breaks this deadlock. The resulting model reaches 6.6 percent word error rate on LibriSpeech test-clean, transcribes in roughly eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages, which we evaluate here on English, Hindi, and Mandarin.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a compelling proof-of-concept for audio-native discrete diffusion language models, effectively solving the gradient flow problem in frozen multimodal adapters through a novel CTC-based grounding strategy, thereby enabling efficient, parallel speech recognition with minimal trainable parameters.
Methodology
The paper proposes a novel architecture for Automatic Speech Recognition (ASR) by adapting a frozen 26B-parameter discrete-diffusion language model (DiffusionGemma) to audio inputs. The core methodological contribution is the "audio-native" pathway, which uses a frozen Whisper encoder, a lightweight trainable projector, and Low-Rank Adapters (LoRA) to inject acoustic features into the LLM's embedding space. A significant technical insight is the diagnosis of the "grounding deadlock," where standard diffusion and autoregressive losses fail to propagate gradients to the audio projector because the frozen attention mechanism ignores the ungrounded audio tokens. The authors resolve this by introducing a Connectionist Temporal Classification (CTC) loss applied directly through the frozen output head, which provides a direct gradient signal to the projector independent of attention. This allows the model to learn to attend to audio features in subsequent steps. The use of uniform random-token diffusion (rather than absorbing masks) is also noted as a specific characteristic of the backbone used.
Results & evaluation
The evaluation is conducted on LibriSpeech (English) and FLEURS (Multilingual: English, Hindi, Mandarin). The model achieves a Word Error Rate (WER) of 6.6% on LibriSpeech test-clean, which is competitive given the extremely low number of trainable parameters (42M, or 0.16% of the backbone). The experiments demonstrate that the model converges in approximately eight parallel denoising steps, decoupling inference cost from transcript length. Ablation studies effectively rule out encoder quality and token resolution as the primary bottlenecks, pointing instead to data scale as the main limitation compared to fully trained autoregressive systems like Whisper. The multilingual results show the adapter's ability to generalize across languages, though performance drops on non-English scripts compared to English.
Reproducibility
The paper provides detailed architectural specifications, including the Whisper encoder size, projector dimensions, LoRA ranks, and training hyperparameters (AdamW, learning rate, batch size). The description of the diffusion process, corruption kernel, and sampling algorithm is sufficiently detailed for reproduction. However, the code is not explicitly linked in the text provided, and the specific random seeds or exact data splits (beyond standard LibriSpeech/FLEURS) are not fully enumerated, though standard practices are implied.
Limitations
The primary limitation is the reliance on a frozen Whisper encoder, which restricts the input window to 30 seconds and prevents true streaming recognition without complex segmentation. The model trails state-of-the-art autoregressive systems (like Whisper-large) in accuracy, particularly on multilingual and noisy data, which the authors attribute to the limited training data (219 hours) compared to the millions of hours used for Whisper. Additionally, the diffusion decoder exhibits repetition artifacts on short utterances, requiring post-processing. The evaluation lacks a controlled head-to-head comparison with other diffusion-based ASR systems on identical hardware and data splits.
Broader impact
This work demonstrates that large language models can be efficiently adapted for speech recognition without retraining the backbone, potentially lowering the barrier to entry for multimodal ASR systems. It challenges the dominance of autoregressive decoding in ASR by showing that parallel diffusion decoding is viable and efficient. The multilingual adapter suggests potential for low-resource language support through parameter-efficient fine-tuning. However, the reliance on pre-trained encoders and large models still poses environmental and accessibility challenges. This paper presents a compelling proof-of-concept for audio-native discrete diffusion language models, effectively solving the gradient flow problem in frozen multimodal adapters through a novel CTC-based grounding strategy, thereby enabling efficient, parallel speech recognition with minimal trainable parameters.
Author of this paper? Send a correction.