INSPIRE: A Benchmark for Instruction-Aware Speech Retrieval
Abstract
Existing speech retrieval systems rely on fixed similarity matching and cannot adapt to diverse user intents. We introduce INSPIRE, the first benchmark for instruction-aware speech retrieval, in which natural-language instructions dynamically specify relevance criteria, including semantic content, speaker identity, speaking style, environmental sounds, and their combinations. We evaluate four retrieval paradigms: large audio-language models, cascaded pipelines, self-supervised speech models, and contrastive audio-language models. Our results reveal that no current method robustly handles all retrieval intents. Text-based approaches perform relatively better at semantic retrieval but struggle with paralinguistic attributes, while speech-based models are moderately better at capturing acoustic properties but falter at following instructions. These findings highlight the need for unified architectures capable of instruction-aware speech retrieval.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
INSPIRE is the first benchmark for instruction-aware speech retrieval, systematically evaluating the ability of various models to follow natural-language instructions for retrieving speech based on semantic, speaker, style, and environmental attributes, revealing a significant gap in current methods' ability to handle multi-attribute, paralinguistic retrieval.
Methodology
The paper proposes INSPIRE, a benchmark for instruction-aware speech retrieval. The methodology involves constructing four subsets (DailyTalk, VCTK, Expresso, Synthetic) to test retrieval across semantic, speaker, style, and environmental attributes. The core technical contribution is the formalization of the problem and the construction of the dataset, particularly the synthetic subset which uses GPT-4o-mini for TTS and GPT-5.2 for instruction generation. The evaluation methodology involves testing four distinct retrieval paradigms: Large Audio-Language Models (LALMs), Cascaded Pipelines (ASR/Captioning + Text Retrieval), Self-Supervised Speech Models, and Contrastive Audio-Language Models. The approach is sound in its comparative design, aiming to highlight the gaps in current models. However, the novelty is somewhat limited by the fact that it is primarily a benchmarking paper rather than proposing a new architecture or algorithm. The use of proprietary models (GPT-4o-mini, GPT-5.2) for data generation raises questions about the independence of the benchmark from specific model biases, although this is common in modern AI research.
Results & evaluation
The experimental evaluation is comprehensive in scope, covering a wide range of existing models and retrieval strategies. The results clearly demonstrate the "modality specialization gap": text-based methods excel at semantic retrieval but fail at paralinguistic tasks, while speech-based models do the opposite. The inclusion of ablation studies on instruction types, pooling strategies, and layer-wise analysis adds depth. The use of Recall@K and NDCG is standard and appropriate. The results are robustly presented with clear tables and radar plots. The finding that no current method handles all intents robustly is significant and well-supported by the data. The synthetic subset's quality assessment (WER, speaker accuracy, emotion accuracy) provides confidence in the dataset's validity.
Reproducibility
The paper provides a GitHub link to the code and data, which is a strong point for reproducibility. The dataset construction process is described in detail, including the sources (DailyTalk, VCTK, Expresso) and the generation pipeline for the synthetic subset. The baseline implementations are standard and well-documented. The use of open-source models for most baselines (HuBERT, WavLM, CLAP, E5-Mistral, etc.) ensures that other researchers can reproduce the experiments. The reliance on proprietary models for some baselines (Gemini, GPT-4o-mini) limits full reproducibility of those specific comparisons but does not invalidate the overall benchmark.
Limitations
A key limitation is the reliance on synthetic data for the most complex multi-attribute tasks. While the synthetic data is high-quality, it may not fully capture the nuances and complexities of real-world speech, such as overlapping speakers, extreme noise, or diverse accents. The instruction generation using GPT-5.2 (a hypothetical or very recent model name, possibly a typo for GPT-4o or similar) introduces a potential bias if the instructions are not diverse enough or if they reflect the specific linguistic patterns of the generator. Additionally, the benchmark is static; it does not address the dynamic nature of user queries in real-world applications. The performance of LALMs is still relatively low, suggesting that the field is in its early stages, which is a limitation of the current state-of-the-art rather than the paper itself, but it highlights the difficulty of the task.
Broader impact
This paper has significant broader impact by defining a new and challenging task in speech processing. It highlights the limitations of current multimodal models and provides a clear direction for future research towards unified, instruction-aware speech retrieval systems. This could lead to more intuitive and powerful speech search engines, accessibility tools, and archival systems. The benchmark will likely spur further research in this area, fostering competition and innovation. The emphasis on paralinguistic attributes (style, emotion, speaker) is particularly important for applications requiring nuanced understanding of human communication. INSPIRE is the first benchmark for instruction-aware speech retrieval, systematically evaluating the ability of various models to follow natural-language instructions for retrieving speech based on semantic, speaker, style, and environmental attributes, revealing a significant gap in current methods' ability to handle multi-attribute, paralinguistic retrieval.
Author of this paper? Send a correction.