MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models
Abstract
While multi-audio understanding is critical for large audio-language models (LALMs), it remains underexplored. We introduce MUGEN, a comprehensive benchmark evaluating this capability across speech, general audio, and music. Our experiments reveal consistent weaknesses in multi-audio settings, and performance degrades sharply as the number of concurrent audio inputs increases, identifying input scaling as a fundamental bottleneck. We further investigate training-free strategies and observe that Audio-Permutational Self-Consistency, which diversifies the order of audio candidates, helps models form more robust aggregated predictions, yielding up to 6.28% accuracy gains. Combining this permutation strategy with Chain-of-Thought further improves performance to 6.74%. These results expose blind spots in current LALMs and provide a foundation for evaluating complex auditory comprehension.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents MUGEN, a comprehensive benchmark for evaluating multi-audio understanding in LALMs, revealing critical blind spots in current models and proposing innovative strategies for improvement. The work is significant as it addresses a largely underexplored area in audio processing, providing a foundation for future advancements in the field.
Methodology
The paper introduces MUGEN, a novel benchmark for multi-audio understanding in large audio-language models (LALMs). The methodology is robust, employing a comprehensive task design that emphasizes cross-audio comparison and integrates various auditory dimensions. The use of training-free strategies like Audio-Permutational Self-Consistency (APSC) is innovative, showcasing a thoughtful approach to overcoming positional biases in audio processing. The combination of self-consistency and permutation strategies represents a significant methodological advancement in the field.
Results & evaluation
The experimental setup is thorough, benchmarking multiple advanced LALMs across diverse tasks. The results clearly demonstrate the limitations of current models in multi-audio scenarios, particularly in non-semantic dimensions. The systematic evaluation of performance degradation with increasing audio inputs is particularly insightful, revealing critical challenges in scaling multi-audio understanding. The use of a variety of models, including proprietary and open-source, adds depth to the comparative analysis.
Reproducibility
The paper provides sufficient detail regarding the experimental setup, including model configurations and evaluation metrics. However, the reliance on proprietary models may limit reproducibility for some aspects of the study. The availability of the MUGEN benchmark on Hugging Face enhances reproducibility for future research.
Limitations
The paper acknowledges that while the proposed benchmark reveals significant weaknesses in current LALMs, it does not address potential solutions for improving non-semantic reasoning capabilities. Additionally, the focus on training-free strategies may overlook the potential benefits of fine-tuning models for enhanced performance.
Broader impact
The introduction of MUGEN has the potential to significantly advance research in multi-audio understanding, paving the way for improved applications in real-world audio-centric tasks. By highlighting the limitations of existing models, the paper encourages further exploration and development of more robust audio-language systems, which could have implications in areas like speech analytics, audio retrieval, and interactive voice agents. This paper presents MUGEN, a comprehensive benchmark for evaluating multi-audio understanding in LALMs, revealing critical blind spots in current models and proposing innovative strategies for improvement. The work is significant as it addresses a largely underexplored area in audio processing, providing a foundation for future advancements in the field.
Author of this paper? Send a correction.