AVID: A Benchmark for Omni-Modal Audio-Visual Inconsistency Understanding via Agent-Driven Construction
Abstract
We present AVID, the first large-scale benchmark for audio-visual inconsistency understanding in videos. While omni-modal large language models excel at temporally aligned tasks such as captioning and question answering, they struggle to perceive cross-modal conflicts, a fundamental human capability that is critical for trustworthy AI. Existing benchmarks predominantly focus on aligned events or deepfake detection, leaving a significant gap in evaluating inconsistency perception in long-form video contexts. AVID addresses this with: (1) a scalable construction pipeline comprising temporal segmentation that classifies video content into Active Speaker, Voiceover, and Scenic categories; an agent-driven strategy planner that selects semantically appropriate inconsistency categories; and five specialized injectors for diverse audio-visual conflict injection; (2) 11.2K long videos (avg. 235.5s) with 39.4K annotated inconsistency events and 78.7K segment clips, supporting evaluation across detection, temporal grounding, classification, and reasoning with 8 fine-grained inconsistency categories. Comprehensive evaluations of state-of-the-art omni-models reveal significant limitations in temporal grounding and reasoning. Our fine-tuned baseline, AVID-Qwen, achieves substantial improvements over the base model (2.8$\times$ higher BLEU-4 in segment reasoning) and surpasses all compared models in temporal grounding (mIoU: 36.1\% vs 26.2\%) and holistic understanding (SODA-m: 7.47 vs 6.15), validating AVID as an effective testbed for advancing trustworthy omni-modal AI systems.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents AVID, the first large-scale benchmark for audio-visual inconsistency understanding in videos, which addresses significant gaps in existing evaluation frameworks. The comprehensive methodology and experimental validation contribute meaningfully to the advancement of trustworthy omni-modal AI systems, highlighting the importance of semantic understanding in audio-visual contexts.
Methodology
The paper introduces a comprehensive methodology for constructing the AVID benchmark, which includes a scalable construction pipeline that utilizes temporal segmentation, an agent-driven strategy planner, and five specialized injectors for audio-visual conflict injection. The methodology is well-structured, addressing the limitations of existing benchmarks by focusing on semantic-level inconsistencies rather than merely artifact detection. The use of a human-in-the-loop process for strategy planning and quality control enhances the reliability of the dataset. The segmentation and classification of video content into distinct categories (Active Speaker, Voiceover, Scenic) is a significant methodological advancement that allows for nuanced evaluation of inconsistencies.
Results & evaluation
The experimental evaluation is robust, featuring a comprehensive analysis of state-of-the-art omni-modal models against the AVID benchmark. The authors provide detailed metrics for detection, temporal grounding, classification, and reasoning tasks, demonstrating clear improvements with their fine-tuned model, AVID-Qwen. The results indicate significant limitations in existing models, particularly in temporal grounding and reasoning, which the authors effectively highlight. The evaluation is thorough, providing both quantitative results and qualitative insights into model performance.
Reproducibility
While the paper outlines a detailed methodology and presents results from various models, it lacks specific URLs or repositories for code and data access, which hinders reproducibility. The implementation details are discussed, but without a public repository, it may be challenging for other researchers to replicate the experiments fully.
Limitations
One limitation is the reliance on a specific set of models for evaluation, which may not encompass the full spectrum of existing omni-modal architectures. Additionally, while the benchmark is extensive, the focus on specific inconsistency categories may limit its applicability to broader contexts. The paper does not address potential biases in the dataset construction process, which could affect the generalizability of the findings.
Broader impact
The AVID benchmark has the potential to significantly advance the field of audio-visual understanding by providing a rigorous testbed for evaluating models on their ability to detect and reason about inconsistencies in video content. This is particularly relevant for applications in trustworthy AI, media verification, and content moderation, where understanding cross-modal relationships is critical. The paper presents AVID, the first large-scale benchmark for audio-visual inconsistency understanding in videos, which addresses significant gaps in existing evaluation frameworks. The comprehensive methodology and experimental validation contribute meaningfully to the advancement of trustworthy omni-modal AI systems, highlighting the importance of semantic understanding in audio-visual contexts.
Author of this paper? Send a correction.