EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning
Abstract
Omni LLMs can naturally locate key audio and visual evidence and integrate information across modalities for audiovisual reasoning. Existing audio-visual reasoning benchmarks that mainly rely on multiple-choice questions usually score models solely based on answer correctness, making it difficult to assess whether a model truly forms a reasoning chain from evidence to the correct answer. In addition, textual or unimodal shortcuts may allow models to answer correctly without analyzing the necessary audio-visual evidence, further weakening the reliability of evaluation benchmarks. To address these issues, we introduce EvaOmni, an evidence-grounded benchmark containing 392 videos and 1,027 purely human-annotated question-answer pairs, covering 15 capabilities and three levels: retrieval and localization, basic understanding, and complex reasoning. We design a series of rigorous annotation procedures in which experts annotate audio and visual evidence and construct video-based distractors; humans review whether questions meet strict inspection criteria, and finally models filter textual and unimodal shortcuts. Beyond high-quality question-answer pairs, we require models to simultaneously return the answer and supporting evidence intervals in order to evaluate the complete reasoning chain. We propose the Coverage Grounded Score (C.G.) to measure how much annotated evidence a model identifies, and the Bottleneck Grounded Score (B.G.) to penalize poor temporal localization in either modality; both are incorporated into answer correctness. Across 7 open-weight and 17 proprietary Omni LLMs, Gemini3.7-Flash achieved the highest overall accuracy (79.4%), C.G. (70.8%), and B.G. (35.1%). At Level 3, it reached 79.1% accuracy, but its B.G. was only 12.3%. These results show that even the most advanced models still have room for development in grounded audio-visual reasoning.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
EvaOmni introduces an evidence-grounded benchmark for audio-visual reasoning that requires models to provide supporting evidence intervals, revealing significant gaps in temporal localization capabilities of current Omni LLMs. The paper makes a strong contribution by addressing the limitations of answer-only evaluation in multimodal benchmarks, proposing rigorous annotation procedures to mitigate shortcuts, and introducing novel metrics (C.G. and B.G.) that assess both answer correctness and evidence grounding. The extensive evaluation across 24 models demonstrates that while top models achieve high accuracy, their ability to precisely localize supporting evidence in both audio and visual modalities remains limited, providing valuable insights for future development of Omni LLMs.
Methodology
The paper proposes EvaOmni, a benchmark that shifts the evaluation paradigm from answer-only correctness to evidence-grounded reasoning. The core methodological contribution is the requirement for models to output timestamped evidence intervals for both audio and visual modalities. The authors introduce two novel metrics: Coverage Grounded Score (C.G.), which measures the overlap of predicted evidence with ground truth, and Bottleneck Grounded Score (B.G.), which penalizes the weaker modality's localization accuracy. The annotation pipeline is rigorous, involving expert annotation, cross-validation, and model-assisted filtering to remove unimodal or textual shortcuts. This approach effectively addresses the "shortcut" problem in existing benchmarks where models can answer correctly without processing the full audio-visual context.
Results & evaluation
The evaluation is extensive, covering 24 Omni LLMs (7 open-weight, 17 proprietary) across 1,027 question-answer pairs. The results clearly demonstrate that while top models like Gemini3.7-Flash achieve high accuracy (79.4%), their grounding scores (B.G. 35.1%) are significantly lower, revealing a gap in precise temporal localization. The ablation studies on input modalities (Audio vs. ASR, Video vs. Audio+Video) provide valuable insights into how models utilize different cues. The comparison with human experts further highlights the difficulty of the task, as humans achieve much higher grounding scores. The cost-efficiency analysis adds practical value for model selection.
Reproducibility
The paper provides detailed descriptions of the annotation process, metric definitions, and experimental setup. The authors state that data, annotations, and evaluation framework are fully released. However, specific URLs for the dataset or code are not explicitly provided in the text snippet, which slightly hinders immediate reproducibility verification, though the claim of release is strong. The prompt templates and preprocessing details are described in the appendix, aiding reproducibility.
Limitations
The benchmark is limited to videos of 1-12 minutes, which may not capture reasoning over longer time horizons. The exclusion of hard-coded subtitles, while preventing OCR shortcuts, limits the diversity of real-world video content. The reliance on multiple-choice questions, even with grounding, may not fully capture open-ended reasoning capabilities. The human baseline allows replay, which is not possible for models, creating a slight asymmetry in comparison conditions.
Broader impact
This work has significant impact on the evaluation of multimodal LLMs. By introducing grounded metrics, it provides a more reliable way to assess whether models truly understand audio-visual content or are relying on shortcuts. This can guide future model development towards better temporal localization and cross-modal integration. The findings that even state-of-the-art models struggle with precise grounding highlight a key area for future research in Omni LLMs. EvaOmni introduces an evidence-grounded benchmark for audio-visual reasoning that requires models to provide supporting evidence intervals, revealing significant gaps in temporal localization capabilities of current Omni LLMs. The paper makes a strong contribution by addressing the limitations of answer-only evaluation in multimodal benchmarks, proposing rigorous annotation procedures to mitigate shortcuts, and introducing novel metrics (C.G. and B.G.) that assess both answer correctness and evidence grounding. The extensive evaluation across 24 models demonstrates that while top models achieve high accuracy, their ability to precisely localize supporting evidence in both audio and visual modalities remains limited, providing valuable insights for future development of Omni LLMs.
Author of this paper? Send a correction.