Omni-DeepSearch: A Benchmark for Audio-Driven Omni-Modal Deep Search
Abstract
Current omni-modal benchmarks mainly evaluate models under settings where multiple modalities are provided simultaneously, while the ability to start from audio alone and actively search for cross-modal evidence remains underexplored. In this paper, we introduce \textbf{Omni-DeepSearch}, a benchmark for audio-driven omni-modal deep search. Given one or more audio clips and a related question, models must infer useful clues from audio, invoke text, image, and video search tools, and perform multi-hop reasoning to produce a short, objective, and verifiable answer. Omni-DeepSearch contains 640 samples across 15 fine-grained categories, covering four retrieval target modalities and four audio content types. A multi-stage filtering pipeline ensures audio dependence, retrieval necessity, visual modality necessity, and answer uniqueness. Experiments on recent closed-source and open-source omni-modal models show that this task remains highly challenging: the strongest evaluated model, Gemini-3-Pro, achieves only 43.44\% average accuracy. Further analyses illustrate key bottlenecks in audio entity inference, query formulation, tool-use reliability, multi-hop retrieval, and cross-modal verification. These results highlight audio-driven omni-modal deep search as an important and underexplored direction for future multimodal agents.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents Omni-DeepSearch, a benchmark for audio-driven omni-modal deep search, highlighting the challenges and limitations of current models while providing a structured methodology for future research in this underexplored area.
Methodology
The paper introduces a novel benchmark, Omni-DeepSearch, which focuses on audio-driven omni-modal deep search, a largely unexplored area in multimodal learning. The methodology is well-structured, with a clear definition of the task and a multi-stage filtering pipeline that ensures the quality and relevance of the dataset. The authors emphasize audio dependence and multi-hop reasoning, which are critical for evaluating models that must infer and retrieve information across different modalities based solely on audio input. The task taxonomy and dataset construction are thorough, providing a solid foundation for future research.
Results & evaluation
The experiments conducted on various models, including both closed-source and open-source, reveal significant challenges in the task, with the best-performing model achieving only 43.44% accuracy. This highlights the complexity of audio-driven retrieval and reasoning, as well as the limitations of current models. The ablation studies and case analyses provide valuable insights into specific failure modes, such as dominant clue bias and misclassification, which are critical for understanding the limitations of existing approaches.
Reproducibility
While the paper provides a comprehensive description of the dataset construction and evaluation metrics, it lacks detailed implementation specifics that would facilitate reproducibility. The absence of a publicly available dataset or code repository further limits the ability of other researchers to replicate the results or build upon this work.
Limitations
The paper acknowledges several limitations, including the inherent ambiguity of audio signals and the reliance on external knowledge for retrieval. Additionally, the performance gap between closed-source and open-source models suggests that there is still much work to be done in improving model capabilities in this domain. The lack of a publicly available dataset or code also hinders broader adoption and experimentation.
Broader impact
The introduction of Omni-DeepSearch has the potential to significantly impact the field of multimodal learning by providing a new benchmark that emphasizes audio as a primary modality for information retrieval. This could lead to advancements in various applications, including voice-activated assistants, audio-based search engines, and enhanced human-computer interaction systems. By addressing the challenges of audio-driven reasoning, this work opens up new avenues for research and development in multimodal AI. The paper presents Omni-DeepSearch, a benchmark for audio-driven omni-modal deep search, highlighting the challenges and limitations of current models while providing a structured methodology for future research in this underexplored area.
Author of this paper? Send a correction.