PitchBench: Measuring Pitch Hearing in Audio-Language Models
Abstract
Audio-language models (ALMs) are increasingly used in real-world applications that require understanding music, from music tutoring and transcription to captioning, recommendation systems, and music production. More broadly, they are becoming an important component of multimodal AI systems that must reason from sensory input rather than text alone. This makes reliable musical perception a critical prerequisite: if a model cannot accurately hear the structure of sound, it cannot be trusted to reason about, teach, transcribe, or act on audio in the real world. Yet existing benchmarks rarely assess one of the most fundamental musical abilities underlying such perception: pitch hearing. Current evaluations tend to probe pitch hearing only indirectly, through higher-level tasks and often in multiple-choice formats, leaving open how reliably ALMs identify fine-grained pitch across instruments, acoustic conditions, and response formats. We introduce PitchBench, an evaluation suite that systematically measures pitch hearing in ALMs. PitchBench comprises 28 experiments spanning absolute and relative pitch perception within sequences and chords, while varying loudness, note duration, sound source, time stretching, background noise, and other acoustic conditions. Tasks range from identifying individual pitches in isolation to tracking a melodic line within a four-part musical texture. Evaluating frontier ALMs, we find that pitch hearing remains highly unreliable: models perform consistently poorly across settings, with accuracy varying sharply by sound source, note duration, and notation format. Current ALMs do not yet possess stable pitch perception, even for controlled synthetic and instrumental stimuli. Alongside the benchmark, we release PitchBench as a Python package containing the evaluation data and data generation tools to support future work on pitch-aware audio-language modeling.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of PitchBench, a systematic evaluation suite for measuring pitch hearing in audio-language models, which significantly enhances the understanding of how these models perceive musical pitch. This work represents a critical step toward improving the reliability and effectiveness of ALMs in real-world audio applications.
Methodology
The methodology presented in PitchBench is robust and systematic, focusing on a hierarchical evaluation of pitch perception in audio-language models (ALMs). The paper introduces a comprehensive framework that includes 28 experiments designed to isolate and assess various aspects of pitch hearing, such as absolute and relative pitch perception. The use of controlled synthetic stimuli allows for precise measurement of model performance across different acoustic conditions and response formats. This structured approach is a significant improvement over existing benchmarks, which often fail to directly evaluate the fundamental ability to perceive pitch.
Results & evaluation
The experimental evaluation is thorough, involving six frontier ALMs across a wide range of tasks that assess pitch perception under varying conditions. The results reveal significant performance variability among models, highlighting specific failure modes that are not captured by higher-level benchmarks. The detailed analysis of model performance, including the effects of acoustic variations and response modalities, provides valuable insights into the strengths and weaknesses of current ALMs in pitch perception.
Reproducibility
The paper emphasizes reproducibility by providing a Python package that includes the evaluation data and generation tools. The authors detail the deterministic generation of stimuli, ensuring that other researchers can replicate the experiments. The inclusion of metadata and standardized output formats further supports reproducibility.
Limitations
While PitchBench offers a significant advancement in evaluating pitch perception, it relies entirely on algorithmically synthesized stimuli, which may not fully capture the complexities of real-world audio. The current instrument selection is limited to General MIDI instruments, and the benchmark does not address non-Western musical traditions or more complex rhythmic reasoning tasks. Future work is needed to incorporate real recordings and broaden the diversity of the instrument pool.
Broader impact
The implications of PitchBench are substantial for the development of audio-language models, particularly in applications requiring reliable musical understanding, such as music tutoring, transcription, and recommendation systems. By providing a diagnostic tool for evaluating pitch perception, this work lays the groundwork for future advancements in multimodal AI systems that integrate audio understanding with other sensory inputs. The main contribution of this paper is the introduction of PitchBench, a systematic evaluation suite for measuring pitch hearing in audio-language models, which significantly enhances the understanding of how these models perceive musical pitch. This work represents a critical step toward improving the reliability and effectiveness of ALMs in real-world audio applications.
Author of this paper? Send a correction.