OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding
Abstract
With the rapid development of multimodal large language models (MLLMs), evaluating their video understanding capabilities has become increasingly important. Existing video understanding benchmarks cover only part of the evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding. We introduce OmniCompass, a holistic benchmark for omni-modal video grounding and understanding. OmniCompass defines two evaluation tracks: OmniCompass-VL assesses vision-language models on tasks requiring visual evidence, while OmniCompass-AV assesses omni-modal models on tasks requiring joint audio-visual evidence. Each track includes three complementary tasks: temporal grounding (TG), spatial grounding (SG), and question answering (QA), assessing when a referenced target appears, where it is, and what happens. QA covers five perception and three reasoning categories in OmniCompass-VL, and three perception and two reasoning categories in OmniCompass-AV. OmniCompass-VL contains 820 videos and 2,460 instances across domains and three duration types: short, medium, and long. A narrative- and audio-rich subset forms OmniCompass-AV with 240 videos and 720 instances. Each instance is evaluated under two paired settings: text-only and text+image, specifying the same target through text and visual-language guidance. All instances are manually annotated with multi-round quality verification. Evaluations of 50 MLLMs show that open-source models still lag behind proprietary counterparts, particularly in audio-visual understanding. Precise temporal and spatial grounding remains challenging even for leading models, and their predictions still exhibit inconsistencies across the paired target specifications.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
OmniCompass introduces a holistic benchmark for omni-modal video understanding that uniquely integrates paired target specifications (text vs. text+image) and complementary evidence tracks (vision-language vs. audio-visual) to diagnose model capabilities in temporal/spatial grounding and question answering. The paper demonstrates rigorous annotation practices and provides extensive evaluations of 50 state-of-the-art models, revealing significant gaps in open-source models' audio-visual integration and cross-reference consistency, thereby offering a critical diagnostic tool for the development of future multimodal large language models.
Methodology
The paper proposes OmniCompass, a benchmark designed to evaluate "omni-modal" video understanding by integrating three key dimensions: target specification (Text vs. Text+Image), evidence modality (Vision-Language vs. Audio-Visual), and task type (Temporal Grounding, Spatial Grounding, Question Answering). The methodology is rigorous, involving a multi-stage annotation process with 1,800 human-hours dedicated to training, calibration, and verification. A notable methodological contribution is the "Cross-Specification Consistency" (CS) metric, which penalizes models not only for incorrect answers but for inconsistencies when the same target is specified via text alone versus text with a reference image. This addresses a gap in existing benchmarks that often treat these as separate tasks rather than testing the model's robustness to reference modality. The construction pipeline uses LLMs (Qwen3-VL) for initial screening but relies heavily on human experts for final annotation, ensuring high-quality ground truth.
Results & evaluation
The experimental section is extensive, evaluating 50 different MLLMs, including both proprietary (Gemini, GPT, Seed, Qwen) and open-source (Kimi, GLM, DeepSeek, LLaVA, etc.) models. The results provide a clear diagnostic picture: proprietary models significantly outperform open-source ones, particularly in audio-visual tasks. The paper provides detailed breakdowns by task (TG, SG, QA), duration (short/medium/long), and domain. The analysis of the impact of timestamps and input modalities (Vision vs. Vision+Audio) is insightful, showing that audio consistently improves QA performance but has variable effects on grounding. The finding that visual references (Text+Image) do not consistently help smaller open-source models is a valuable insight for the community.
Reproducibility
The paper provides a strong reproducibility statement, detailing the annotation protocols, data sources (including licenses), and evaluation metrics. The benchmark data is described as being available, though specific URLs are not explicitly listed in the provided text snippet (often found in the appendix or project page). The detailed description of the annotation pipeline and the use of public datasets for the base videos enhances reproducibility. The code for evaluation metrics (CS) is likely available given the standard practices for such benchmark papers.
Limitations
The primary limitation is the scale of the audio-visual subset (OmniCompass-AV), which contains only 240 videos compared to 820 in the VL track. This smaller sample size may limit the statistical power of the AV-specific conclusions. Additionally, the benchmark relies on a specific set of 50 models; while comprehensive, the rapid evolution of MLLMs means the leaderboard will become outdated quickly. The "Text+Image" setting requires careful curation to ensure the image doesn't leak the answer, which is a non-trivial annotation challenge acknowledged in the paper.
Broader impact
This benchmark is highly relevant to the current trend of developing omni-modal models (processing text, image, video, and audio simultaneously). By providing a unified framework for evaluating grounding and understanding across these modalities, it will likely become a standard reference for evaluating next-generation MLLMs. The focus on cross-specification consistency is a novel angle that pushes models to be more robust to how targets are referenced, which is crucial for real-world applications like video search and retrieval. OmniCompass introduces a holistic benchmark for omni-modal video understanding that uniquely integrates paired target specifications (text vs. text+image) and complementary evidence tracks (vision-language vs. audio-visual) to diagnose model capabilities in temporal/spatial grounding and question answering. The paper demonstrates rigorous annotation practices and provides extensive evaluations of 50 state-of-the-art models, revealing significant gaps in open-source models' audio-visual integration and cross-reference consistency, thereby offering a critical diagnostic tool for the development of future multimodal large language models.
Author of this paper? Send a correction.