SonicBench: Dissecting the Physical Perception Bottleneck in Large Audio Language Models
Abstract
Large Audio Language Models (LALMs) excel at semantic and paralinguistic tasks, yet their ability to perceive the fundamental physical attributes of audio such as pitch, loudness, and spatial location remains under-explored. To bridge this gap, we introduce SonicBench, a psychophysically grounded benchmark that systematically evaluates 12 core physical attributes across five perceptual dimensions. Unlike previous datasets, SonicBench uses a controllable generation toolbox to construct stimuli for two complementary paradigms: recognition (absolute judgment) and comparison (relative judgment). This design allows us to probe not only sensory precision but also relational reasoning capabilities, a domain where humans typically exhibit greater proficiency. Our evaluation reveals a substantial deficiency in LALMs' foundational auditory understanding; most models perform near random guessing and, contrary to human patterns, fail to show the expected advantage on comparison tasks. Furthermore, explicit reasoning yields minimal gains. However, our linear probing analysis demonstrates crucially that frozen audio encoders do successfully capture these physical cues (accuracy at least 60%), suggesting that the primary bottleneck lies in the alignment and decoding stages, where models fail to leverage the sensory signals they have already captured.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of SonicBench, a benchmark that systematically evaluates the physical perception capabilities of LALMs, revealing critical deficiencies in their auditory understanding and suggesting pathways for future research. The study's innovative methodology and rigorous experimental design provide valuable insights into the limitations of current models and highlight the importance of foundational auditory understanding in advancing the field.
Methodology
The paper introduces SonicBench, a benchmark designed to evaluate the physical perception capabilities of Large Audio Language Models (LALMs). The methodology is innovative as it combines psychophysics with machine learning, allowing for a nuanced understanding of LALMs' auditory processing. The use of a controllable generation toolbox to create stimuli for both recognition and comparison tasks is particularly noteworthy, as it enables a comprehensive assessment of sensory precision and relational reasoning. However, the reliance on linear classifiers for probing may limit the depth of insights into model capabilities.
Results & evaluation
The experiments are well-structured, focusing on 12 core physical attributes across five perceptual dimensions. The results indicate a significant gap in LALMs' performance compared to human benchmarks, highlighting a critical area for future research. The finding that frozen audio encoders can capture physical cues with at least 60% accuracy is promising, suggesting that the issue lies in the alignment and decoding stages rather than the sensory input itself. However, the evaluation could be strengthened by including more diverse datasets and exploring multilingual contexts.
Reproducibility
The paper lacks detailed implementation specifics, which could hinder reproducibility. While the methodology is described, the absence of code or a project URL limits the ability of other researchers to replicate the findings. Providing access to the dataset and the generation toolbox would enhance reproducibility and facilitate further research in this area.
Limitations
The paper acknowledges several limitations, including the focus on English text instructions, which may not reflect the performance of LALMs in multilingual settings. Additionally, the controlled nature of the dataset may restrict acoustic variability, potentially affecting the generalizability of the results. The authors also note that while their probing methodology is effective, it may not capture the full complexity of LALMs' auditory processing capabilities.
Broader impact
The findings of this study have significant implications for the development of more robust audio processing models. By identifying the physical perception bottleneck in LALMs, the research paves the way for future advancements in audio language models that can better understand and interpret the fundamental attributes of sound. This could lead to improvements in various applications, including speech recognition, music analysis, and interactive audio systems. The main contribution of this paper is the introduction of SonicBench, a benchmark that systematically evaluates the physical perception capabilities of LALMs, revealing critical deficiencies in their auditory understanding and suggesting pathways for future research. The study's innovative methodology and rigorous experimental design provide valuable insights into the limitations of current models and highlight the importance of foundational auditory understanding in advancing the field.
Author of this paper? Send a correction.