ALM2Vec: Learning Audio Embeddings for Universal Audio Retrieval with Large Audio-Language Models
Abstract
Recent advances in language--audio retrieval have been largely driven by contrastive dual-encoder architectures that align audio and text in a shared embedding space. While effective, existing retrieval embeddings are primarily optimized for audio--caption matching, limiting their ability to support diverse retrieval objectives and controllable retrieval behaviors. We present ALM2Vec, a universal audio embedding framework derived from pretrained large audio--language models (LALMs). By transferring the audio understanding, instruction-following, and reasoning capabilities acquired through large-scale multimodal training, ALM2Vec learns a unified embedding space for retrieval across audio domains and task types. Beyond conventional text--audio retrieval, ALM2Vec incorporates natural-language instructions into the embedding process, enabling instruction-aware retrieval for scenarios such as audio question answering and aspect-conditioned retrieval. Experimental results show that ALM2Vec achieves competitive performance on standard audio and speech retrieval benchmarks while exhibiting promising compositional and controllable retrieval capabilities, highlighting its potential as a unified audio embedding model for retrieval across domains, tasks, and user intents.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
ALM2Vec presents a compelling adaptation of Large Audio-Language Models for universal audio retrieval, achieving competitive performance on standard benchmarks and demonstrating unique instruction-aware capabilities, though it faces challenges regarding computational efficiency and the trade-off between retrieval optimization and general reasoning.
Methodology
The paper proposes ALM2Vec, a framework that adapts Large Audio-Language Models (LALMs), specifically MiDashengLM, for universal audio retrieval. The core methodology involves freezing the audio encoder and applying LoRA to the LLM component, then extracting the final [EOS] token's hidden state as the embedding representation. This is projected into a fixed-dimensional space and trained with a bidirectional contrastive loss. The novelty lies in leveraging the instruction-following and reasoning capabilities of LALMs to create "instruction-aware" embeddings, allowing for controllable retrieval (e.g., retrieving based on specific acoustic attributes or questions) rather than just holistic semantic matching. While the approach of adapting LLMs for embeddings is not entirely new (e.g., LLM2Vec), applying it to the audio domain with a focus on instruction-conditioned retrieval is a meaningful extension. However, the technical innovation is incremental, relying on standard contrastive learning and LoRA adaptation.
Results & evaluation
The evaluation covers three main areas: Audio-Text Retrieval (AudioCaps, Clotho), Speech-Text Retrieval (LibriSQA), and Audio Question Answering (MMAU-mini). 1. **Audio-Text:** ALM2Vec-FT achieves competitive results on AudioCaps and Clotho, outperforming strong CLAP baselines on Clotho, which contains longer, more complex audio. This supports the claim of better long-range dependency modeling. 2. **Speech-Text:** On LibriSQA, ALM2Vec-FT significantly outperforms CLAP and even the cascaded Whisper+BGE pipeline, demonstrating strong semantic speech understanding without explicit ASR training. This is a strong result. 3. **QA:** On MMAU-mini, ALM2Vec-PT performs competitively with large multimodal models, but fine-tuning for retrieval actually hurts performance, suggesting a trade-off between retrieval alignment and general reasoning. The experiments are well-conducted and cover relevant benchmarks. The inclusion of instruction-following case studies adds qualitative value, showing the model can distinguish between hard negatives based on specific instructions.
Reproducibility
The paper provides sufficient detail on the model architecture (MiDashengLM backbone, LoRA config), training stages (pretraining vs. fine-tuning), and loss functions. The use of open-source datasets (AudioCaps, Clotho, LibriSQA, MMAU) ensures reproducibility. The release of code/project page further aids reproducibility.
Limitations
1. **Performance Trade-off:** The drop in QA performance after retrieval fine-tuning suggests that optimizing for retrieval similarity may degrade the model's broader reasoning capabilities. 2. **Latency/Compute:** Using a large LLM backbone for embedding extraction is computationally expensive compared to dedicated dual-encoder models like CLAP, which may limit real-time applications. 3. **Instruction Sensitivity:** While promising, the instruction-following capability is demonstrated via case studies rather than rigorous quantitative benchmarks for "controllable retrieval," making it hard to gauge the robustness of this feature at scale. 4. **Audio Length:** The fine-tuning audio length is limited to 30 seconds, which may restrict performance on very long-form audio despite the backbone's capability.
Broader impact
ALM2Vec contributes to the growing field of multimodal foundation models by demonstrating that LALMs can serve as effective universal embedding backends. The ability to perform instruction-aware retrieval has significant implications for accessible media search, content-based recommendation systems, and audio data curation. It moves beyond simple caption matching to more nuanced, user-intent-driven retrieval. ALM2Vec presents a compelling adaptation of Large Audio-Language Models for universal audio retrieval, achieving competitive performance on standard benchmarks and demonstrating unique instruction-aware capabilities, though it faces challenges regarding computational efficiency and the trade-off between retrieval optimization and general reasoning.
Author of this paper? Send a correction.