Pruned CTC for Memory-Efficient Large-Vocabulary ASR Training
Abstract
Connectionist temporal classification (CTC) naturally supports offline and streaming speech recognition with utterance-level supervision, but conventional implementations materialize frame-by-vocabulary activations in memory, making CTC training with native LLM vocabularies prohibitively memory-intensive. A key observation is that every valid CTC alignment uses only target tokens and blank, and their union across a batch typically forms a small subset of the full vocabulary. We introduce Pruned CTC, which restricts alignment computation to this subset while retaining full-vocabulary normalization. We prove that this vocabulary reduction is exactly equivalent to full-vocabulary CTC in loss and gradients. Head-and-loss activation memory no longer scales linearly with vocabulary size. We further apply finite-beam alignment pruning. Building on Pruned CTC, we develop LLM-CTC, which adapts pretrained LLMs for non-autoregressive ASR while retaining causal attention and native vocabularies, and extend it to bounded-history streaming, avoiding chunk-level speech--text alignments. Experiments show that, with Zipformer-M encoder and 180K vocabulary, Pruned CTC reduces full-step memory by 5.1$\times$ with only 17% step-time overhead. Across three corpora, it matches standard CTC accuracy. On GigaSpeech, across six Qwen3 model sizes from 0.6B to 32B, LLM-CTC remains within 7% relative WER of LLM-CE with 7 to 10$\times$ faster recognition; when fine-tuning Qwen3-ASR for bounded-history streaming, LLM-CTC remains within 3% relative WER of matched offline models on the test set. Together, these results establish Pruned CTC as a scalable sequence objective for native-vocabulary LLM ASR across offline and streaming settings.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Pruned CTC enables memory-efficient training of large-vocabulary ASR models by exploiting the sparsity of CTC alignments, and LLM-CTC successfully adapts pretrained LLMs for fast, non-autoregressive speech recognition with strong performance in both offline and streaming settings.
Methodology
The paper introduces "Pruned CTC," a memory-efficient implementation of Connectionist Temporal Classification (CTC) that exploits the sparsity of valid alignments. The core insight is that while the softmax normalization requires the full vocabulary, the dynamic programming alignment only involves the target tokens and the blank symbol. By restricting the alignment computation to this small subset while retaining full-vocabulary normalization via a "complement" class, the authors prove exact equivalence in loss and gradients. This is a mathematically sound and clever optimization. The method is further extended to "LLM-CTC," which adapts pretrained Large Language Models (LLMs) for non-autoregressive ASR. This is a significant architectural contribution, as it allows leveraging the linguistic knowledge of LLMs without the latency of autoregressive decoding, and it handles streaming via bounded-history attention masks without requiring chunk-level alignments.
Results & evaluation
The experiments are robust and cover a wide range of model sizes (0.6B to 32B Qwen3 models) and datasets (GigaSpeech, etc.). The results demonstrate a 5.1x reduction in memory with minimal time overhead. The comparison against LLM-CE (Cross-Entropy) shows that LLM-CTC achieves comparable accuracy (within 7% relative WER) while being 7-10x faster in recognition. The streaming results are particularly strong, showing that the method avoids the complex alignment requirements of previous streaming LLM-ASR methods. The inclusion of both offline and streaming evaluations provides a comprehensive view of the method's utility.
Reproducibility
The authors provide a GitHub repository link and state that code and pretrained models will be open-sourced. The paper includes detailed algorithmic descriptions (Algorithm 1) and mathematical proofs, which aids reproducibility. The specific hyperparameters for the alignment beam and chunking are provided in the appendices (referenced).
Limitations
The method relies on the assumption that the union of target tokens in a batch is small relative to the full vocabulary, which holds for typical ASR tasks but might not for extremely diverse or noisy data. The "finite-beam" approximation introduces a small error, though the paper claims it is negligible ($10^{-11}$). The streaming extension requires careful management of KV caches and attention masks, which can be complex to implement in production systems.
Broader impact
This work has high potential impact on the deployment of LLM-based ASR systems. By enabling non-autoregressive decoding with LLMs, it significantly reduces inference latency and computational cost, making real-time, high-quality speech recognition more accessible. The memory efficiency also allows for training larger models or larger batches on standard hardware. This could accelerate the adoption of LLMs in speech processing pipelines. Pruned CTC enables memory-efficient training of large-vocabulary ASR models by exploiting the sparsity of CTC alignments, and LLM-CTC successfully adapts pretrained LLMs for fast, non-autoregressive speech recognition with strong performance in both offline and streaming settings.
Author of this paper? Send a correction.