Scaling Audio-Text Retrieval with Multimodal Large Language Models
Abstract
Audio-text retrieval is crucial for bridging acoustic signals and natural language. While contrastive dual-encoder architectures like CLAP have shown promise, they are fundamentally limited by the capacity of small-scale encoders. Specifically, the text encoders struggle to understand complex queries that require reasoning or world knowledge. In this paper, we propose AuroLA, a novel contrastive language-audio pre-training framework that re-purposes Multimodal Large Language Models (MLLMs) as a unified backbone for retrieval. Specifically, we make three contributions: (i) we construct a scalable data pipeline that curates diverse audio from multiple sources and generates multi-granular captions, ranging from long descriptions to structured tags, via automated annotation; (ii) we adapt an MLLM for retrieval by prompting it to summarize the audio/text input and using the hidden state of a special token as audio/text embeddings. For model training, we devise a novel Hybrid-NCE loss, which employs multi-granular supervision and hard-negative reweighting to robustly align audio with diverse textual supervision; and (iii) we design an MLLM-based bidirectional re-ranking module that refines retrieval candidates through deep cross-modal interaction. Extensive experiments demonstrate that AuroLA consistently outperforms state-of-the-art models, including the recent PE-AV, while utilizing only approximately 1% of PE-AV's training data. Lastly, we observe clear scaling trends regarding dataset size and model capacity, validating the effectiveness of MLLM as a unified backbone for audio-text retrieval. Code is available at https://github.com/Jazzcharles/AuroLA.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of AuroLA, a novel framework that effectively utilizes Multimodal Large Language Models for audio-text retrieval, demonstrating significant improvements over existing methods. The comprehensive analysis of the technical contributions, innovative methodology, and promising experimental results highlight its potential impact on the field of machine learning and audio processing.
Methodology
The proposed AuroLA framework introduces a novel approach to audio-text retrieval by leveraging Multimodal Large Language Models (MLLMs) as a unified backbone. The methodology is well-structured, with a focus on creating a scalable data pipeline and a Hybrid-NCE loss that enhances the alignment of audio and text embeddings through multi-granular supervision. The adaptation of MLLMs for retrieval tasks is innovative, particularly the use of a special token's hidden state for embeddings. However, the paper could benefit from a more detailed explanation of the implementation of the Hybrid-NCE loss and its advantages over traditional contrastive losses.
Results & evaluation
The experiments conducted are extensive, demonstrating the superiority of AuroLA over existing state-of-the-art models, including PE-AV, while using significantly less training data. The results are compelling, showcasing clear scaling trends that validate the proposed framework. However, the paper lacks a thorough comparison with a broader range of models and datasets, which could provide a more comprehensive understanding of AuroLA's performance across different scenarios.
Reproducibility
The paper mentions that code is available on GitHub, which is a positive aspect for reproducibility. However, the paper does not provide sufficient implementation details or hyperparameter settings that would allow other researchers to easily replicate the experiments. A more detailed supplementary material or appendix could enhance reproducibility.
Limitations
One limitation is the reliance on the quality and diversity of the audio data curated for training, which may affect the generalizability of the model. Additionally, while the use of MLLMs is innovative, the computational cost associated with training and deploying such models could be a barrier to practical applications. The paper also does not address potential biases in the data or the model's performance across different languages or dialects.
Broader impact
The implications of this research are significant, particularly in applications such as multimedia search engines, accessibility tools for the hearing impaired, and content-based audio retrieval systems. By improving audio-text retrieval capabilities, this work could enhance user experiences in various domains, including education, entertainment, and information retrieval. The main contribution of this paper is the introduction of AuroLA, a novel framework that effectively utilizes Multimodal Large Language Models for audio-text retrieval, demonstrating significant improvements over existing methods. The comprehensive analysis of the technical contributions, innovative methodology, and promising experimental results highlight its potential impact on the field of machine learning and audio processing.
Author of this paper? Send a correction.