OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning
Abstract
Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio--visual reasoning, creating an urgent need for evaluations that expose their capability limits. Audio--visual captioning is an ideal diagnostic task, yet current benchmarks face a coupled trade-off: whole-caption scores provide coverage without localization, local probes provide localization without coverage, and unconstrained LLM judges introduce instability. We introduce OmniCapBench (Omni-Video Caption Benchmark), a benchmark that reframes audio--visual caption evaluation as a deep-structured diagnostic framework. OmniCapBench shifts the prediction target from free-form text to sets of atomic, verifiable evaluation units across three tracks: entity references, visual shots, and audio events, enabling reliable scoring with deterministic constraint checks and localized LLM-based semantic comparisons. With 786 densely annotated videos, OmniCapBench effectively distinguishes MLLM perception errors, including temporal grounding failures, identity drift, cross-modal misalignment, and hallucinated descriptions. Evaluating frontier MLLMs reveals strong local perception but weak long-horizon audio--visual reasoning, particularly in identity drift and cross-modal misalignment, providing a fine-grained roadmap for omnimodal development.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces OmniCapBench, a deep-structured benchmark that reframes audio-visual captioning evaluation by decomposing free-form text into atomic, verifiable units (entities, shots, events), enabling reliable, fine-grained diagnostic scoring that exposes specific model failures like identity drift and temporal grounding errors which holistic metrics obscure.
Methodology
The paper proposes a "deep-structured" evaluation framework that shifts the paradigm from scoring free-form text captions to scoring structured, atomic JSON units (References, Shots, Events). This is a methodologically sound approach to solving the "opaque score" problem in LLM-as-a-Judge evaluations. By decoupling structural verification (deterministic rules for IDs, timestamps, and links) from semantic verification (bounded LLM checks on aligned units), the authors effectively mitigate the instability and hallucination sensitivity of global LLM judges. The construction pipeline, which uses a multi-model ensemble (Generator, Evaluator, Refiner, Selector) with strict schema validation, is robust and well-designed to ensure ground-truth integrity. The bidirectional evaluation (Precision vs. Recall) for open-vocabulary fields is a clever addition that prevents models from gaming the metric via verbosity or terseness.
Results & evaluation
The experimental section is rigorous and provides strong evidence for the benchmark's utility. The authors evaluate 9 state-of-the-art omnimodal models, distinguishing between proprietary and open-source variants. The results clearly demonstrate the benchmark's ability to differentiate models that might appear similar under holistic metrics. The "Auditing the Legacy Evaluation Paradigm" section is particularly strong, providing empirical proof that global LLM judges are unstable (scores varying by >20 points based on judge strength) and that holistic scores mask local errors. The error analysis, decomposing failures into specific structural violations (e.g., identity drift, temporal grounding), offers actionable insights for model developers.
Reproducibility
The paper provides high reproducibility. It details the exact JSON schema, the filtering thresholds for data curation, the temperature settings for inference, and the specific prompts used for generation and judging (referenced in the appendix). The project website is provided, which likely hosts the dataset and code. The clear separation of deterministic rules from LLM-based scoring makes the evaluation pipeline easy to replicate and debug.
Limitations
The primary limitation is the reliance on structured JSON output for evaluation. This may not reflect the natural usage of MLLMs, which are often deployed for free-form conversation or captioning. Forcing models to output strict JSON can introduce a separate mode of failure (formatting errors) that is distinct from perceptual errors, although the authors do account for this with the SGC (Schema Gate Check) metric. Additionally, the benchmark is limited to videos under 5 minutes, which may not capture long-horizon reasoning capabilities in longer films or lectures. The annotation process, while rigorous, is still automated/semi-automated, and potential biases in the "Evaluator" models used to construct the ground truth could propagate into the benchmark.
Broader impact
This benchmark has significant potential to standardize the evaluation of omnimodal models. By providing a fine-grained diagnostic tool, it helps researchers identify specific bottlenecks (e.g., cross-modal alignment vs. entity tracking) rather than just overall performance. This granularity is crucial for the next generation of audio-visual AI, where understanding the *relationship* between sound and sight is key. The framework could be extended to other multimodal tasks (e.g., video QA, action recognition) by adapting the atomic unit structure. The paper introduces OmniCapBench, a deep-structured benchmark that reframes audio-visual captioning evaluation by decomposing free-form text into atomic, verifiable units (entities, shots, events), enabling reliable, fine-grained diagnostic scoring that exposes specific model failures like identity drift and temporal grounding errors which holistic metrics obscure.
Author of this paper? Send a correction.