HumMusQA: A Human-written Music Understanding QA Benchmark Dataset
Abstract
The evaluation of music understanding in Large Audio-Language Models (LALMs) requires a rigorously defined benchmark that truly tests whether models can perceive and interpret music, a standard that current data methodologies frequently fail to meet. This paper introduces a meticulously structured approach to music evaluation, proposing a new dataset of 320 hand-written questions curated and validated by experts with musical training, arguing that such focused, manual curation is superior for probing complex audio comprehension. To demonstrate the use of the dataset, we benchmark six state-of-the-art LALMs and additionally test their robustness to uni-modal shortcuts.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces HumMusQA, a novel benchmark dataset for evaluating music understanding in LALMs, emphasizing expert curation and rigorous question design to enhance the assessment of audio comprehension. The comprehensive methodology and experimental framework contribute significantly to the field, paving the way for more effective evaluations of music AI systems.
Methodology
The paper presents a robust methodology for developing a music understanding QA benchmark, emphasizing the importance of expert curation in question design. The authors utilize a structured approach that combines expert knowledge with a diverse selection of audio tracks, ensuring comprehensive coverage of musical characteristics. The iterative peer review process for question validation enhances the dataset's quality, addressing common pitfalls in automated question generation. However, the reliance on expert authorship may limit scalability and introduce subjectivity in question difficulty.
Results & evaluation
The experiments benchmark six state-of-the-art LALMs, providing a clear evaluation framework that includes randomized answer ordering to mitigate bias. The results indicate varying performance across models, highlighting strengths in cultural reasoning over formal music theory tasks. The systematic analysis of model responses offers insights into their reasoning mechanisms, although the overall performance remains moderate. The dataset's design allows for meaningful comparisons across different models, contributing to the field's understanding of LALM capabilities in music comprehension.
Reproducibility
The dataset is publicly available under Creative Commons licenses, facilitating reproducibility. The authors provide detailed descriptions of their methodology, including audio sourcing and question design processes. However, the reproducibility of model evaluations may depend on access to the specific models used, which could be a barrier for some researchers.
Limitations
The study acknowledges potential weaknesses in the human-authored questions, particularly regarding their vulnerability to being answered based solely on textual cues. Additionally, the dataset's reliance on expert knowledge may limit its applicability to broader contexts or casual listeners. The performance drop in music theory questions suggests that the benchmark may not fully capture the complexity of music understanding.
Broader impact
This work has significant implications for advancing music AI research, providing a high-quality benchmark that can drive improvements in LALMs' music comprehension capabilities. The dataset can serve as a foundation for future studies exploring audio understanding, reasoning, and the integration of music theory in AI systems. By addressing the perception gap in current benchmarks, this research encourages the development of models that genuinely understand music rather than relying on superficial linguistic cues. The paper introduces HumMusQA, a novel benchmark dataset for evaluating music understanding in LALMs, emphasizing expert curation and rigorous question design to enhance the assessment of audio comprehension. The comprehensive methodology and experimental framework contribute significantly to the field, paving the way for more effective evaluations of music AI systems.
Author of this paper? Send a correction.