PianoBench: Evaluating Musical Perception and Reasoning in Language Models
Abstract
Multimodal language models continue to improve on broad music understanding benchmarks, but which musical features can they perceive and reason about? We investigate this question by introducing PianoBench, a benchmark that tasks language models with reasoning about musical performances across congruent acoustic and symbolic input representations. PianoBench is derived from our new multimodal, training-scale dataset comprising more than 20,000 hours of interleaved transcripts of music and spoken musical commentary from instructional piano videos. Open-weight audio-language models achieve at most 52.8% accuracy on PianoBench, and at best only modestly outperform their accuracy on a no-audio control. In contrast, frontier models using chain of thought achieve up to 93.2% accuracy when provided transcripts of music as text, exposing a gap between perceiving musical features from audio and reasoning about musical events when they are given explicitly. We then test whether mid-training on interleaved musical instruction can improve question answering without chain of thought, raising Qwen 3.5 9B Base's zero-shot accuracy on PianoBench with music transcripts from 42.8% to 56.0% after instruction tuning. These experiments suggest that instructional videos may represent an overlooked source of multimodal supervision for music. Available: https://gclef-cmu.org/pianobench.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces PianoBench, a benchmark and dataset for evaluating musical perception and reasoning in language models, revealing a significant gap between audio perception and symbolic reasoning capabilities in current models. Through a rigorous data pipeline and comprehensive experiments, the authors demonstrate that while frontier models excel at reasoning over explicit note events, open-weight models struggle with audio perception, and that interleaved musical instruction can improve zero-shot reasoning without chain-of-thought.
Methodology
The paper introduces a novel data pipeline to construct a large-scale (20k hours) dataset of interleaved musical commentary and symbolic transcripts from instructional piano videos. The methodology is robust, utilizing source separation, speech recognition, and MIDI transcription to create aligned multimodal data. The benchmark construction is rigorous, employing LLMs for question generation and distractor creation, followed by human verification to ensure quality. The proposed "MIDI-text" tokenization scheme is a clever contribution, allowing text-only LLMs to process symbolic music efficiently. The experimental design effectively isolates perception (audio) from reasoning (symbolic) by comparing performance across different input modalities.
Results & evaluation
The evaluation is comprehensive, testing a wide range of open-weight and proprietary models. The key finding—that open-weight models struggle with audio perception while frontier models excel at symbolic reasoning with CoT—is well-supported by the data. The mid-training experiment on Qwen 3.5 9B Base provides strong evidence that interleaved instruction data improves zero-shot reasoning without CoT, outperforming separated data. The inclusion of a no-music control is crucial for validating the necessity of the musical input.
Reproducibility
High. The authors release the dataset, benchmark, and code. The data pipeline is described in detail, including the specific models used for transcription and separation. The hyperparameters for mid-training are provided. The use of specific commercial APIs (GPT 5.6, Gemini 3.1) limits full reproducibility for those specific results, but the open-weight model results are reproducible.
Limitations
The benchmark is limited to piano music, which may not generalize to other instruments or genres. The reliance on YouTube instructional videos may introduce biases in repertoire and teaching style. The "MIDI-text" representation, while efficient, is a lossy compression of the original audio and may not capture timbral or expressive nuances. The small size of the benchmark (250 questions) limits statistical power for fine-grained analysis.
Broader impact
This work highlights a critical gap in current audio-language models: the ability to reason about music when given symbolic representations versus perceiving it from audio. It provides a valuable resource (PianoTalk dataset) for training models on fine-grained musical understanding. The findings suggest that current open-weight models need better supervision for audio perception, while frontier models' reasoning capabilities can be leveraged to generate better training data. This could drive future research in multimodal music understanding and the development of more perceptually grounded audio models. The paper introduces PianoBench, a benchmark and dataset for evaluating musical perception and reasoning in language models, revealing a significant gap between audio perception and symbolic reasoning capabilities in current models. Through a rigorous data pipeline and comprehensive experiments, the authors demonstrate that while frontier models excel at reasoning over explicit note events, open-weight models struggle with audio perception, and that interleaved musical instruction can improve zero-shot reasoning without chain-of-thought.
Author of this paper? Send a correction.