MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition
Abstract
Large language models (LLMs) have recently shown strong potential in audio-visual speech recognition (AVSR), but their high computational demands and sensitivity to token granularity limit their practicality in resource-constrained settings. Token compression methods can reduce inference cost, but they require fixing a compression rate in advance and produce a single fixed-length output, offering no flexibility to balance information density and efficiency at inference time. Matryoshka representation learning (MRL) addresses this by enabling a single model to operate across multiple token granularities, allowing compression rates to be adjusted dynamically. However, current MRL-based methods treat each scale independently during training, limiting cross-scale generalization, robustness at high compression, and interpretability. To overcome these limitations, we propose MoME (Mixture of Matryoshka Experts), a novel framework that integrates sparse Mixture-of-Experts (MoE) into MRL-based LLMs for AVSR. MoME augments a frozen LLM with top-k routed and shared experts, allowing dynamic capacity allocation across scales and modalities. A shared router promotes consistent expert activation across granularities, enabling compressed sequences to benefit from representations learned at lower compression. Experiments on LRS2 and LRS3 demonstrate that MoME achieves state-of-the-art performance across AVSR, ASR, and VSR tasks, while requiring significantly fewer parameters and maintaining robustness under noise. MoME unifies the adaptability of MRL with the efficiency of MoE, offering a scalable and interpretable solution for resource-aware speech recognition.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of MoME, a novel framework that effectively combines Mixture-of-Experts with Matryoshka representation learning to enhance audio-visual speech recognition. This work is significant as it addresses key challenges in the field, such as computational efficiency and robustness, while providing a scalable solution that can be adapted to various multimodal tasks. The methodology is well-founded, and the experimental results support the claims made, although the lack of reproducibility and statistical significance reporting could be improved.
Methodology
The proposed MoME framework integrates a Mixture-of-Experts (MoE) architecture with Matryoshka representation learning (MRL) to enhance audio-visual speech recognition (AVSR). This approach is innovative as it allows for dynamic capacity allocation across different token granularities, addressing the limitations of existing MRL methods that treat scales independently. The incorporation of a shared router for expert activation across scales is a notable design choice that promotes cross-scale generalization and robustness, which is well-justified in the paper. The methodology is sound and shows a clear understanding of the challenges in AVSR.
Results & evaluation
The experiments conducted on LRS2 and LRS3 datasets demonstrate the effectiveness of MoME in achieving state-of-the-art performance across various tasks (AVSR, ASR, VSR) while maintaining fewer parameters. The paper provides a comprehensive analysis of different configurations and ablation studies, which strengthens the validity of the results. However, the lack of statistical significance reporting for the experimental results is a missed opportunity for deeper insight into the robustness of the findings.
Reproducibility
The paper includes sufficient details on training and testing protocols, including data preprocessing, training hyperparameters, and model configurations. However, the absence of publicly available code or a clear path for reproducibility limits the practical application of the results. The authors mention that the code will be available upon acceptance, which is a positive step.
Limitations
The paper acknowledges that while MoE models reduce inference-time computation, they still require all experts to reside in memory, which can lead to increased memory usage. Additionally, the inference cost is noted to be slightly higher than some baselines, which could be a concern for real-world applications. The focus on audio-visual models also limits the generalizability of the findings to other domains.
Broader impact
The paper discusses potential societal impacts, particularly the risks associated with deploying LLMs that may produce biased or inaccurate outputs. This acknowledgment is crucial, as it highlights the need for careful evaluation before practical applications. The authors suggest conducting thorough safety and fairness evaluations, which is a responsible approach to mitigate potential negative consequences. The main contribution of this paper is the introduction of MoME, a novel framework that effectively combines Mixture-of-Experts with Matryoshka representation learning to enhance audio-visual speech recognition. This work is significant as it addresses key challenges in the field, such as computational efficiency and robustness, while providing a scalable solution that can be adapted to various multimodal tasks. The methodology is well-founded, and the experimental results support the claims made, although the lack of reproducibility and statistical significance reporting could be improved.
Author of this paper? Send a correction.