Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Models
Abstract
Large language models (LLMs) have recently achieved impressive results in speech recognition across multiple modalities, including Auditory Speech Recognition (ASR), Visual Speech Recognition (VSR), and Audio-Visual Speech Recognition (AVSR). Despite this progress, current LLM-based approaches typically address each task independently, training separate models that raise computational and deployment resource use while missing potential cross-task synergies. They also rely on fixed-rate token compression, which restricts flexibility in balancing accuracy with efficiency. These limitations highlight the need for a unified framework that can support ASR, VSR, and AVSR while enabling elastic inference. To this end, we present Omni-AVSR, a unified audio-visual LLM that combines efficient multi-granularity training with parameter-efficient adaptation. Specifically, we adapt the matryoshka representation learning paradigm to efficiently train across multiple audio and visual granularities, reducing its inherent training resource use. Furthermore, we explore three LoRA-based strategies for adapting the backbone LLM, balancing shared and task-specific specialization. Experiments on LRS2 and LRS3 show that Omni-AVSR achieves comparable or superior accuracy to state-of-the-art baselines while training a single model at substantially lower training and deployment resource use. The model also remains robust under acoustic noise, and we analyze its scaling behavior as LLM size increases, providing insights into the trade-off between performance and efficiency.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the development of Omni-AVSR, a unified audio-visual LLM that effectively integrates multiple speech recognition modalities while optimizing for resource efficiency. This work significantly advances the field by addressing the limitations of existing models and proposing a novel methodology that balances performance and efficiency in multimodal speech recognition tasks.
Methodology
The paper introduces a unified framework, Omni-AVSR, which leverages a matryoshka representation learning paradigm to facilitate multi-granularity training across ASR, VSR, and AVSR tasks. This approach is innovative as it aims to reduce resource consumption while maintaining performance, addressing a significant gap in the current literature where these tasks are treated independently. The use of LoRA-based strategies for parameter-efficient adaptation is also a notable contribution, allowing for shared and task-specific model specialization.
Results & evaluation
The experiments conducted on the LRS2 and LRS3 datasets are comprehensive, demonstrating that Omni-AVSR achieves comparable or superior accuracy to existing state-of-the-art models while significantly lowering resource use. The robustness of the model under acoustic noise and the analysis of scaling behavior as LLM size increases provide valuable insights into performance-efficiency trade-offs, enhancing the credibility of the results.
Reproducibility
The paper provides a GitHub repository link for code access, which is essential for reproducibility. However, further details on the experimental setup, hyperparameters, and specific configurations used in the training process would enhance reproducibility.
Limitations
One limitation is the reliance on fixed-rate token compression, which may still impose constraints on flexibility despite the proposed improvements. Additionally, the paper does not extensively discuss the potential challenges in deploying the unified model in real-world scenarios, particularly in diverse acoustic environments.
Broader impact
The unified approach to multimodal speech recognition has the potential to streamline applications in various fields, including human-computer interaction, accessibility technologies, and automated transcription services. By reducing resource requirements, it could lead to more efficient deployment of speech recognition systems in resource-constrained environments. The main contribution of this paper is the development of Omni-AVSR, a unified audio-visual LLM that effectively integrates multiple speech recognition modalities while optimizing for resource efficiency. This work significantly advances the field by addressing the limitations of existing models and proposing a novel methodology that balances performance and efficiency in multimodal speech recognition tasks.
Author of this paper? Send a correction.