Can Large Audio Language Models Ignore Multilingual Distractors? An Evaluation of Their Selective Auditory Attention Capabilities
Abstract
Robust selective auditory attention under multilingual interference is critical for reliable deployment of Large Audio Language Models (LALMs). We introduce MUSA, a cocktail party-inspired multilingual benchmark for source-grounded spoken-language understanding and reasoning. Each item pairs an English target dialogue with a semantically plausible distractor in English, Spanish, Korean, or Chinese, and evaluates models across (1) single, (2) source separation-based two-stage, (3) and end-to-end cocktail party settings under controlled SNRs. Evaluating two closed-source and four open-weight LALMs, we find that strong single performance does not ensure robust selective auditory attention: cocktail party accuracy degrades under severe SNRs, and errors are dominated by distractor-grounded source confusion. In addition, separation reduces acoustic overlap but leaves source attribution unresolved, often yielding confident wrong-stream answers. Data and code will be released upon publication.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of the MUSA benchmark, which evaluates the selective auditory attention of LALMs in multilingual contexts, revealing critical insights into their performance limitations. This work significantly advances the understanding of LALMs' capabilities and highlights the importance of robust auditory attention mechanisms in real-world applications.
Methodology
The paper introduces a novel benchmark, MUSA, designed to evaluate the selective auditory attention capabilities of Large Audio Language Models (LALMs) in the presence of multilingual distractors. The methodology is well-structured, employing a cocktail party paradigm that mimics real-world scenarios where multiple languages may interfere with audio processing. The authors rigorously define the experimental settings, including single, separation-based, and cocktail party conditions, and provide a detailed diagnostic error taxonomy that categorizes model failures. This structured approach allows for a comprehensive understanding of the models' performance under varying signal-to-noise ratios (SNRs), which is a significant advancement in the field.
Results & evaluation
The experiments are robust, involving six different LALMs evaluated across multiple settings and SNR levels. The results clearly demonstrate that high performance in single-stream conditions does not translate to effective performance in cocktail party scenarios, highlighting a critical gap in current LALM capabilities. The authors provide detailed statistical analyses and error distributions, which enhance the validity of their findings. The use of synthesized audio ensures consistency, although it may limit ecological validity.
Reproducibility
The paper mentions that data and code will be released upon publication, which is a positive aspect for reproducibility. However, the specifics of the implementation details, such as the exact configurations of the models and the separation techniques used, could be better documented to facilitate independent replication of the results.
Limitations
The study is limited by the relatively small dataset of 200 synthesized cases, which may not capture the full variability of natural speech. Additionally, the focus on English as the target language and the use of a single off-the-shelf separator may restrict the generalizability of the findings. The authors also acknowledge potential confounding factors that could influence cross-lingual understanding, which are not fully controlled.
Broader impact
The findings have significant implications for the deployment of LALMs in high-stakes environments such as healthcare and aviation, where accurate audio understanding is critical. By addressing the challenges of multilingual interference, this research paves the way for more reliable audio processing systems that can operate effectively in diverse linguistic contexts. The introduction of MUSA as a benchmark could stimulate further research in this area, leading to advancements in model architectures and training methodologies. The main contribution of this paper is the introduction of the MUSA benchmark, which evaluates the selective auditory attention of LALMs in multilingual contexts, revealing critical insights into their performance limitations. This work significantly advances the understanding of LALMs' capabilities and highlights the importance of robust auditory attention mechanisms in real-world applications.
Author of this paper? Send a correction.