Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical Scores
Abstract
Understanding complete musical scores requires reasoning over symbolic structures such as pitch, rhythm, harmony, and form. Despite the rapid progress of Large Language Models (LLMs) and Vision-Language Models (VLMs) in natural language and multimodal tasks, their ability to comprehend musical notation remains underexplored. We introduce Musical Score Understanding Benchmark (MSU-Bench), the first large-scale, human-curated benchmark for evaluating score-level musical understanding across both textual (ABC notation) and visual (PDF) modalities. MSU-Bench comprises 1,800 generative question-answer (QA) pairs drawn from works spanning Bach, Beethoven, Chopin, Debussy, and others, organised into four progressive levels of comprehension: Onset Information, Notation & Note, Chord & Harmony, and Texture & Form. Through extensive zero-shot and fine-tuned evaluations of over 15+ state-of-the-art (SOTA) models, we reveal sharp modality gaps, fragile level-wise success rates, and the difficulty of sustaining multilevel correctness. Fine-tuning markedly improves performance in both modalities while preserving general knowledge, establishing MSU-Bench as a rigorous foundation for future research at the intersection of Artificial Intelligence (AI), musicological, and multimodal reasoning.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents the Musical Score Understanding Benchmark (MSU-Bench), a pioneering effort to evaluate large language models' comprehension of musical scores, highlighting significant gaps in current models' abilities and establishing a foundation for future research in AI and music. The combination of textual and visual modalities, along with a structured assessment framework, marks a notable contribution to the field, although further details on methodology and reproducibility could enhance its impact.
Methodology
The paper introduces a novel benchmark, MSU-Bench, which is a significant advancement in evaluating LLMs and VLMs in the context of musical score understanding. The methodology is well-structured, with a clear delineation of comprehension levels that allows for a comprehensive assessment of models. The approach of combining both textual and visual modalities is innovative, addressing a gap in existing research. However, the paper could benefit from a more detailed explanation of the criteria for selecting the QA pairs and the rationale behind the progressive levels of comprehension.
Results & evaluation
The experiments are robust, involving a wide range of state-of-the-art models and thorough evaluations in both zero-shot and fine-tuned settings. The results highlight significant modality gaps and the challenges of achieving multilevel correctness, providing valuable insights into the capabilities of current models. However, the paper could improve by including more quantitative metrics and comparisons with baseline models to strengthen the findings.
Reproducibility
The paper lacks detailed implementation specifics, such as the exact configurations used for the fine-tuning of models and the preprocessing steps for the musical scores. This omission may hinder reproducibility efforts by other researchers. Including a supplementary material or a dedicated section with these details would enhance the paper's reproducibility.
Limitations
One limitation is the reliance on a specific set of composers, which may not fully represent the diversity of musical styles and complexities. Additionally, the benchmark may not account for the nuances of musical interpretation, which could affect the generalizability of the results. The paper also does not address potential biases in the dataset or the models used.
Broader impact
The establishment of MSU-Bench has the potential to significantly impact the fields of AI and musicology by providing a standardized framework for evaluating musical understanding in AI systems. This could lead to advancements in music generation, analysis, and education tools, fostering greater interaction between AI and the arts. The research opens avenues for interdisciplinary collaboration and could inspire further exploration into multimodal AI applications. The paper presents the Musical Score Understanding Benchmark (MSU-Bench), a pioneering effort to evaluate large language models' comprehension of musical scores, highlighting significant gaps in current models' abilities and establishing a foundation for future research in AI and music. The combination of textual and visual modalities, along with a structured assessment framework, marks a notable contribution to the field, although further details on methodology and reproducibility could enhance its impact.
Author of this paper? Send a correction.