MRMAD: A Multi-Round Multi-Audio Benchmark for Evaluating Acoustic Degradation Perception in Large Audio-Language Models
Abstract
Large audio-language models (LALMs) have shown promising progress in understanding speech, music, and general sound events, yet their ability to reason about how audio signals are degraded remains underexplored. Existing benchmarks primarily evaluate semantic understanding, event recognition, or high-level audio reasoning, leaving a basic question unanswered: Do LALMs understand the differences in audio quality? We introduce MRMAD, a Multi-Round Multi-Audio Degradation benchmark for evaluating audio degradation perception and understanding in LALMs. MRMAD spans speech, music, and sound, and frames evaluation as multi-turn dialogues over multiple audio inputs, requiring models to identify degradation types, compare severity, and perceive corruption changes across turns. Unlike current single-turn audio-language benchmarks, MRMAD evaluates whether LALMs can maintain consistent degradation hypotheses with new evidence and explain low-level acoustic phenomena in natural language. Through a systematic evaluation of 18 representative LALMs from non-thinking to reasoning and Omni models, we find that current models often recognize coarse content while failing to diagnose, compare, or reason about degradations reliably. MRMAD reveals an important yet overlooked aspect of audio-language understanding and provides a diagnostic foundation for building future LALMs that are robust to real-world acoustic conditions.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
[The paper presents MRMAD, a novel multi-turn, multi-audio benchmark that systematically evaluates the ability of Large Audio-Language Models to perceive, compare, and reason about audio degradations across speech, music, and sound domains, revealing significant limitations in current models' low-level acoustic understanding].
Methodology
The paper introduces MRMAD, a benchmark designed to evaluate Large Audio-Language Models (LALMs) on their ability to perceive and reason about audio degradation. The methodology involves constructing a dataset of 8,400 multiple-choice questions across three domains (speech, music, sound) and nine degradation types (e.g., noise, reverberation, compression). The evaluation is framed as multi-turn dialogues where models must identify degradation types, compare severity between two clips, or rank three clips by severity. The approach is methodologically sound for its purpose: it isolates low-level perceptual capabilities from high-level semantic understanding by using clean references and controlled degradations. The multi-turn design is a clever solution to the problem of comparing multiple audio inputs without temporal localization confusion. However, the novelty is moderate as it is essentially a specialized benchmark addition to the growing list of audio evaluation suites, rather than proposing a new model architecture or fundamental algorithm.
Results & evaluation
The experimental setup is comprehensive, evaluating 18 representative LALMs, including open-source models (SALMONN, Qwen2-Audio, Audio Flamingo) and closed-source/reasoning models (GPT-4o, Gemini, Qwen3-Omni). The results clearly demonstrate that current state-of-the-art models perform near random chance on degradation type identification and struggle significantly with severity comparison and ranking. This provides a valuable diagnostic baseline. The evaluation covers a wide range of model sizes and types, strengthening the generalizability of the findings. The use of balanced sampling and specific distractor generation for multiple-choice questions adds rigor to the evaluation protocol.
Reproducibility
The paper provides a clear data construction pipeline, specifying source datasets (VCTK, LibriSpeech, MUSDB18, etc.), degradation parameters, and question generation templates. The code and dataset are made available via a GitHub link, which significantly enhances reproducibility. The multi-turn format and template-based prompt generation are described in sufficient detail for others to replicate the evaluation.
Limitations
A key limitation is the reliance on synthetic degradations applied to high-quality source audio. While this allows for controlled severity levels, it may not fully capture the complexity of real-world audio degradation, which often involves complex, non-linear interactions between noise sources and signal characteristics. Additionally, the multiple-choice format, while efficient for large-scale evaluation, may not fully capture the nuanced reasoning capabilities of models compared to open-ended generation tasks. The exclusion of certain degradations for specific domains (e.g., echo in music) is justified but limits the universality of the benchmark. Finally, the evaluation is limited to 18 models; as the field evolves rapidly, the benchmark will need continuous updating.
Broader impact
This work has significant implications for the development of robust audio-language models. By highlighting the gap in low-level audio perception, it directs future research towards improving model robustness to real-world acoustic conditions. This is crucial for applications in accessibility (e.g., audio description for visually impaired users in noisy environments), audio restoration, and high-fidelity communication systems. The benchmark serves as a necessary tool for the community to track progress in this underexplored area. [The paper presents MRMAD, a novel multi-turn, multi-audio benchmark that systematically evaluates the ability of Large Audio-Language Models to perceive, compare, and reason about audio degradations across speech, music, and sound domains, revealing significant limitations in current models' low-level acoustic understanding].
Author of this paper? Send a correction.