MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMs
Abstract
Audiovisual arts encompass diverse creative disciplines, including cinema, visual arts, stage performance, and game design, where artistic meaning arises from deliberate combinations of visual, auditory, and narrative elements (e.g., fear amplified through claustrophobic framing, or grief conveyed through silence and lingering close-ups). True artistic understanding extends beyond recognizing what is depicted to reasoning about why it is expressed through particular creative choices. Despite the strong progress of multimodal large language models (MLLMs), this critical aspect of artistic understanding remains underexplored, as existing benchmarks largely measure perceptual recognition while overlooking reasoning about creative intent. To address this gap, we introduce Musebench, a comprehensive benchmark designed to evaluate MLLMs on nuanced artistic understanding. It comprises 4,016 questions spanning cinematic arts, static visual arts, stage performing arts, and game arts, distilled from over 10K candidate video essays that pair professional commentary with visual demonstration. To capture the open-ended nature of artistic analysis at scale, the benchmark combines single-select and variable-option multi-select questions. All questions are generated and refined through a four-phase iterative pipeline combining shortcut filtering, adversarial distractors, and expert validation. Comprehensive zero-shot evaluation of 28 state-of-the-art MLLMs reveals that even the best-performing model achieves only 48.29% accuracy, substantially below human expert performance of 87.18%, exposing a significant gap in current models' creative domain expertise.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents MuseBench, a comprehensive and rigorously constructed benchmark for evaluating intent-level audiovisual arts understanding in MLLMs, addressing a critical gap in current multimodal evaluation by focusing on interpretive reasoning rather than mere perceptual recognition, and providing valuable insights into the limitations of state-of-the-art models through extensive zero-shot evaluation and principled metrics.
Methodology
The paper introduces MuseBench, a novel benchmark designed to evaluate Multimodal Large Language Models (MLLMs) on intent-level audiovisual arts understanding. The methodology is rigorous and addresses a significant gap in existing benchmarks, which typically focus on perceptual recognition rather than interpretive reasoning. Key methodological contributions include: (1) Leveraging video essays as a scalable source of expert-narrated knowledge, using a four-phase pipeline (segmentation, captioning, question generation, distractor generation) to create high-quality QA pairs. (2) Introducing a mixed-format question design (single-select and multi-select) to capture the plurality of artistic interpretation. (3) Developing a principled evaluation protocol with Chance-Adjusted Accuracy (CAA) for single-select and set-based F1 for multi-select questions, which mitigates biases from variable option counts and partial credit. The use of adversarial distractors and iterative human-in-the-loop review enhances the robustness of the benchmark.
Results & evaluation
The authors conduct a comprehensive zero-shot evaluation of 28 state-of-the-art MLLMs, including proprietary (GPT-5.4, Claude-4.6-Opus, etc.) and open-source models (Qwen, InternVL, LLaVA, VideoLLaMA, etc.). The results reveal a substantial performance gap between the best models (approx. 48% CAA) and human experts (87% accuracy), highlighting the current limitations of MLLMs in artistic reasoning. The evaluation includes detailed analyses of per-category performance, modality contributions, key-frame selection efficacy, and positional bias. The findings are insightful, particularly the observation that game arts are a shared weakness and that open-source models exhibit a pronounced first-position bias. The ablation studies on modality and key-frame selection provide valuable insights into model capabilities and limitations.
Reproducibility
The paper provides detailed descriptions of the data collection and annotation pipeline, including specific prompts and failure taxonomies. The benchmark statistics and evaluation metrics are clearly defined. The project page URL is provided, suggesting that code and data may be available, which supports reproducibility. The iterative review process and expert validation steps are well-documented, adding to the transparency of the benchmark construction.
Limitations
The benchmark is limited to four art categories (cinematic, static visual, stage performing, game arts) and relies on video essays as the primary data source, which may not fully capture the diversity of artistic expression in raw creative works. The availability of video essays varies across art forms and languages, potentially introducing bias. The evaluation is restricted to multiple-choice formats, which may not fully capture the open-ended nature of artistic analysis. Future work should explore expanding to additional art forms and developing open-ended evaluation methods.
Broader impact
MuseBench provides a valuable tool for assessing and improving the artistic understanding capabilities of MLLMs. It highlights the need for models to go beyond perceptual recognition and engage in deeper interpretive reasoning. The benchmark can guide future research in multimodal learning, particularly in integrating domain-specific expertise and cultural priors. It also raises important questions about the role of human expertise in AI evaluation and the potential for AI to assist in artistic analysis and education. This paper presents MuseBench, a comprehensive and rigorously constructed benchmark for evaluating intent-level audiovisual arts understanding in MLLMs, addressing a critical gap in current multimodal evaluation by focusing on interpretive reasoning rather than mere perceptual recognition, and providing valuable insights into the limitations of state-of-the-art models through extensive zero-shot evaluation and principled metrics.
Author of this paper? Send a correction.