A Language-Agnostic Hierarchical LoRA-MoE Architecture for CTC-based Multilingual ASR
Abstract
Large-scale multilingual ASR (mASR) models such as Whisper achieve strong performance but incur high computational and latency costs, limiting their deployment on resource-constrained edge devices. In this study, we propose a lightweight and language-agnostic multilingual ASR system based on a CTC architecture with domain adaptation. Specifically, we introduce a Language-agnostic Hierarchical LoRA-MoE (HLoRA) framework integrated into an mHuBERT-CTC model, enabling end-to-end decoding via LID-posterior-driven LoRA routing. The hierarchical design consists of a multilingual shared LoRA for learning language-invariant acoustic representations and language-specific LoRA experts for modeling language-dependent characteristics. The proposed routing mechanism removes the need for prior language identity information or explicit language labels during inference, achieving true language-agnostic decoding. Experiments on MSR-86K and the MLC-SLM 2025 Challenge datasets demonstrate that HLoRA achieves competitive performance with state-of-the-art two-stage inference methods using only single-pass decoding, significantly improving decoding efficiency for low-resource mASR applications.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a significant advancement in multilingual ASR by introducing a lightweight, language-agnostic framework that effectively balances efficiency and performance. The comprehensive analysis of the technical contributions, methodology, and experimental validation underscores its potential impact on the field.
Methodology
The paper introduces a novel Language-Agnostic Hierarchical LoRA-MoE architecture that leverages a combination of shared and language-specific LoRA modules to enhance multilingual ASR performance. The hierarchical design is well-structured, allowing for efficient single-pass decoding without the need for explicit language identification during inference. The integration of LID-posterior-driven routing is a significant innovation, as it simplifies the inference process while maintaining competitive accuracy. The methodology is theoretically sound and addresses key challenges in multilingual ASR, particularly in low-resource settings.
Results & evaluation
The experiments are conducted on well-established datasets (MSR-86K and MLC-SLM 2025 Challenge), providing a robust basis for evaluation. The results demonstrate that the proposed HLoRA framework achieves competitive performance compared to state-of-the-art two-stage inference methods while significantly improving decoding efficiency. The ablation studies further validate the design choices made in the architecture, showcasing the trade-offs between shared and language-specific representations.
Reproducibility
The authors have made their models and source code publicly available, which is critical for reproducibility. However, the paper lacks detailed implementation specifics, such as hyperparameter settings and training procedures, which could aid other researchers in replicating the results more effectively.
Limitations
One limitation is the potential trade-off between efficiency and accuracy, as indicated by the slight performance degradation when using shared LoRA modules. Additionally, the reliance on a specific pretrained model (mHuBERT-147) may limit the generalizability of the findings to other architectures or datasets. The paper could also benefit from a more thorough discussion of the computational costs associated with the proposed methods.
Broader impact
The proposed framework has significant implications for real-world applications of multilingual ASR, particularly in resource-constrained environments such as mobile devices and hearing aids. By enabling efficient language-agnostic decoding, this work could facilitate broader access to ASR technologies across diverse linguistic contexts, enhancing communication and information accessibility. The paper presents a significant advancement in multilingual ASR by introducing a lightweight, language-agnostic framework that effectively balances efficiency and performance. The comprehensive analysis of the technical contributions, methodology, and experimental validation underscores its potential impact on the field.
Author of this paper? Send a correction.