Text-Prompted CLAP: Learning Query-Conditioned Audio Representations via Contrastive Learning
Abstract
Contrastive Language-Audio Pretraining (CLAP) learns aligned text and audio representations in a shared embedding space. However, independent encoding of each modality limits its ability to model cross-modal semantics in complex audio understanding and retrieval tasks. To address this limitation, this paper proposes Text-Prompted CLAP (TP-CLAP), a parameter-efficient extension of CLAP that introduces a cross-attention-based fusion module to incorporate textual prompts into audio features. TP-CLAP is trained using an audio multiple-choice question (audio-MCQ) framework, where it learns to align query-conditioned audio representations with text embeddings of correct answer choices via contrastive learning. Experiments demonstrate that TP-CLAP achieves competitive performance on audio question answering (audio-QA) with substantially larger audio-LLMs, while also improving the base CLAP model on audio-text retrieval and zero-shot classification benchmarks. The learned representations are further fine-tuned for attribute-focused audio-to-audio retrieval, showing that TP-CLAP consistently outperforms the standard CLAP baseline in music retrieval tasks.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a method for learning query-conditioned audio representations by integrating a cross-attention fusion module into the CLAP framework, trained via an audio multiple-choice question paradigm, demonstrating significant improvements in attribute-focused retrieval and competitive performance in audio question answering tasks.
Methodology
The paper proposes TP-CLAP, a parameter-efficient extension of the CLAP model that introduces a cross-attention-based fusion module to condition audio representations on textual prompts. The core methodological contribution lies in the training framework: it leverages AudioMCQ data to train the model to align query-conditioned audio embeddings with the text embeddings of correct multiple-choice answers. This is combined with standard CLAP contrastive loss and auxiliary losses to maintain general alignment. The approach is technically sound, addressing a specific limitation of static embeddings in CLAP (inability to handle query-dependent tasks like QA or attribute-focused retrieval). The use of cross-attention to inject textual context into audio features is a known pattern (similar to Q-Former in LLaVA or cross-attention in BLIP), but its application specifically to enable attribute-focused audio retrieval via an MCQ framework is a distinct and practical contribution. The derivation of the loss functions, including the specific handling of multi-label attributes in the fine-tuning stage, is rigorous.
Results & evaluation
The experimental evaluation is comprehensive, covering audio-text retrieval, zero-shot classification, audio question answering (QA), and attribute-focused audio-to-audio retrieval. The results show consistent improvements over the base CLAP model and the CLAP+AudioMCQ baseline (which lacks the fusion module). Notably, TP-CLAP achieves competitive performance on audio QA against much larger audio-LLMs, which is a strong result highlighting the efficiency of the approach. The attribute-focused retrieval experiments are particularly compelling, demonstrating that the model can isolate specific attributes (e.g., instrument, genre) rather than relying on global similarity. The use of standard benchmarks (AudioCaps, Clotho, ESC50, FSD50K, NSynth, MagnaTagATune) adds credibility. However, the improvement in standard retrieval tasks is modest, suggesting the primary value is in the conditional/attribute-specific capabilities.
Reproducibility
The paper provides detailed implementation details, including encoder architectures (CED-Base, bert-base-uncased), hyperparameters (learning rates, batch sizes, warmup epochs), and loss function weights. The data sources (AudioMCQ, AudioCaps, Clotho, WavCaps, MACS, NSynth, MagnaTagATune) are well-known and accessible. The training stages are clearly delineated. This level of detail allows for reasonable reproducibility, assuming access to the pre-trained CLAP weights and the specified datasets.
Limitations
The paper acknowledges that the model is smaller than audio-LLMs like Audio Flamingo 3, which still outperform it in QA. The improvements on standard retrieval tasks are marginal. The method relies heavily on the quality of the AudioMCQ dataset; if the questions/answers are noisy or biased, the model may learn spurious correlations. Furthermore, the cross-attention module adds computational overhead during inference compared to the original CLAP, although it remains lightweight compared to full LLMs. The paper does not discuss potential biases in the attribute-focused retrieval (e.g., gender or cultural biases in instrument/genre associations).
Broader impact
This work contributes to the field of multimodal audio understanding by enabling more flexible and query-dependent audio representation learning. It facilitates applications such as advanced music information retrieval (MIR), where users might search for "sad piano music" or "jazz with saxophone," rather than just general genre or mood. It also improves accessibility tools that need to understand specific audio contexts. The parameter-efficient nature makes it more accessible for deployment on edge devices or with limited compute compared to large multimodal LLMs. This paper presents a method for learning query-conditioned audio representations by integrating a cross-attention fusion module into the CLAP framework, trained via an audio multiple-choice question paradigm, demonstrating significant improvements in attribute-focused retrieval and competitive performance in audio question answering tasks.
Author of this paper? Send a correction.