Spatial Blind Spot: Auditory Motion Perception Deficits in Audio LLMs
Abstract
Large Audio-Language Models (LALMs) have recently shown impressive progress in speech recognition, audio captioning, and auditory question answering. Yet, whether these models can perceive spatial dynamics, particularly the motion of sound sources, remains unclear. In this work, we uncover a systematic motion perception deficit in current ALLMs. To investigate this issue, we introduce AMPBench, the first benchmark explicitly designed to evaluate auditory motion understanding. AMPBench introduces a controlled question-answering benchmark designed to evaluate whether Audio-Language Models (LALMs) can infer the direction and trajectory of moving sound sources from binaural audio. Comprehensive quantitative and qualitative analyses reveal that current models struggle to reliably recognize motion cues or distinguish directional patterns. The average accuracy remains below 50%, underscoring a fundamental limitation in auditory spatial reasoning. Our study highlights a fundamental gap between human and model auditory spatial reasoning, providing both a diagnostic tool and new insight for enhancing spatial cognition in future Audio-Language Models.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of AMPBench, a benchmark for evaluating auditory motion perception in Audio-Language Models, revealing significant deficits in current models' abilities to understand spatial dynamics in sound. This work is a critical step toward improving auditory cognition in machine learning, highlighting the need for further research in this area.
Methodology
The paper introduces AMPBench, a novel benchmark specifically designed to evaluate auditory motion perception in Audio-Language Models (LALMs). The methodology is well-structured, focusing on controlled question-answering tasks that assess models' abilities to infer sound source motion from binaural audio. The introduction of a benchmark for this specific aspect of auditory perception is a significant contribution, as it addresses a gap in existing evaluation frameworks. However, the paper could benefit from a more detailed description of the benchmark's design and the criteria used for evaluating model performance.
Results & evaluation
The experiments conducted are comprehensive, involving both quantitative and qualitative analyses of various LALMs. The results indicate a clear performance gap, with models achieving less than 50% accuracy in recognizing motion cues. This finding is critical as it highlights a fundamental limitation in current models. However, the paper lacks a detailed comparison of different models and their specific weaknesses, which could provide deeper insights into the challenges faced in auditory motion perception.
Reproducibility
The paper does not provide sufficient details regarding the implementation of the benchmark or the models tested, which raises concerns about reproducibility. Including access to datasets, model architectures, and evaluation scripts would enhance the reproducibility of the findings and allow other researchers to validate the results.
Limitations
The study acknowledges limitations in the current models' performance but does not explore potential reasons for these deficits in depth. Additionally, the benchmark may not cover all aspects of auditory motion perception, limiting its applicability. The lack of a demo or project URL also restricts further exploration of the proposed methods.
Broader impact
This research has significant implications for the development of more sophisticated audio processing models, particularly in applications such as virtual reality, robotics, and assistive technologies for the hearing impaired. By identifying and addressing the limitations in auditory motion perception, the findings could guide future research directions and model enhancements. The main contribution of this paper is the introduction of AMPBench, a benchmark for evaluating auditory motion perception in Audio-Language Models, revealing significant deficits in current models' abilities to understand spatial dynamics in sound. This work is a critical step toward improving auditory cognition in machine learning, highlighting the need for further research in this area.
Author of this paper? Send a correction.