MuseAgent-1: Interactive Grounded Multimodal Understanding of Music Scores and Performance Audio
Abstract
Despite recent advances in multimodal large language models (MLLMs), their ability to understand and interact with music remains limited. Music understanding requires grounded reasoning over symbolic scores and expressive performance audio, which general-purpose MLLMs often fail to handle due to insufficient perceptual grounding. We introduce MuseAgent, a music-centric multimodal agent that augments language models with structured symbolic representations derived from sheet music images and performance audio. By integrating optical music recognition and automatic music transcription modules, MuseAgent enables multi-step reasoning and interaction over fine-grained musical content. To systematically evaluate music understanding capabilities, we further propose MuseBench, a benchmark covering music theory reasoning, score interpretation, and performance-level analysis across text, image, and audio modalities. Experiments show that existing MLLMs perform poorly on these tasks, while MuseAgent achieves substantial improvements, highlighting the importance of structured multimodal grounding for interactive music understanding.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of MuseAgent, a multimodal agent designed to enhance music understanding by integrating symbolic representations from sheet music and performance audio. This work is significant as it addresses a notable gap in the capabilities of existing multimodal large language models (MLLMs) regarding music, a domain that requires complex reasoning across different modalities. The novelty lies in the structured approach to grounding music understanding, which is not commonly explored in the current literature. The technical impact is substantial, as it not only proposes a new framework but also establishes a benchmark (MuseBench) for evaluating music understanding, potentially guiding future research in this area. Overall, the paper contributes meaningfully to the field of machine learning by expanding the application of MLLMs to a complex and underexplored domain.
Author of this paper? Send a correction.