Decoder-only Conformer with Modality-aware Sparse Mixtures of Experts for ASR
Abstract
We present a decoder-only Conformer for automatic speech recognition (ASR) that processes speech and text in a single stack without external speech encoders or pretrained large language models (LLM). The model uses a modality-aware sparse mixture of experts (MoE): disjoint expert pools for speech and text with hard routing and top-1 selection, embedded in hybrid-causality Conformer blocks (bidirectional for speech, causal for text). Training combines CTC on speech positions with label-smoothed cross-entropy for text generation. Our 113M-parameter model consistently improves WER over a 139M AED baseline on Librispeech (2.8% vs. 3.2% test-clean; 5.6% vs. 6.0% test-other). On Common Voice 16.1 with a single multilingual model across five languages, our approach reduces average WER from 12.2% to 10.6%. To our knowledge, this is the first randomly initialized decoder-only ASR that surpasses strong AED baselines via modality-aware routing and sparse MoE, achieving better accuracy with fewer active parameters and without alignment/adaptation modules.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a decoder-only Conformer architecture that effectively integrates modality-aware sparse mixtures of experts for automatic speech recognition. The innovative approach and solid experimental results position it as a valuable contribution to the field, although further work is needed to enhance reproducibility and address practical deployment challenges.
Methodology
The paper introduces a novel decoder-only Conformer architecture that integrates modality-aware sparse mixtures of experts (MoE) for automatic speech recognition (ASR). The methodology is well-structured, leveraging a single stack to process both speech and text without the need for external encoders or pretrained models. The use of disjoint expert pools for speech and text, along with hard routing and top-1 selection, is innovative and addresses the challenge of heterogeneous modality integration effectively. The hybrid causality approach is also a significant contribution, allowing for bidirectional processing of speech while maintaining causal generation for text. However, the paper could benefit from a more detailed explanation of the routing mechanism and its implications on model performance.
Results & evaluation
The experiments are robust, demonstrating consistent improvements in word error rates (WER) over strong baselines across multiple datasets, including Librispeech and Common Voice 16.1. The results validate the proposed model's effectiveness, showing that it can outperform traditional encoder-decoder architectures while maintaining a lower parameter count. The comparative analysis against various baselines is thorough, but additional ablation studies could further clarify the contributions of individual components, such as the modality-aware routing and load-balancing loss.
Reproducibility
The paper provides sufficient implementation details, including model configurations, training epochs, and data augmentation techniques, which facilitate reproducibility. However, the absence of a publicly available code repository or demo limits the ability for other researchers to replicate the results independently. Including a link to the code would significantly enhance the paper's reproducibility.
Limitations
While the proposed model shows promising results, it relies on a relatively complex architecture that may pose challenges in practical deployment scenarios, especially in real-time applications. Additionally, the paper does not address the potential computational overhead introduced by the MoE mechanism, which may affect inference speed. Future work should also consider the scalability of the model to larger datasets and more diverse languages.
Broader impact
The research has significant implications for the field of automatic speech recognition, particularly in unifying speech and text processing within a single framework. This could lead to more efficient and effective ASR systems, especially in multilingual contexts. The approach may also inspire further research into modality-aware architectures in other domains, such as natural language processing and computer vision. This paper presents a decoder-only Conformer architecture that effectively integrates modality-aware sparse mixtures of experts for automatic speech recognition. The innovative approach and solid experimental results position it as a valuable contribution to the field, although further work is needed to enhance reproducibility and address practical deployment challenges.
Author of this paper? Send a correction.