ISA-Bench: Benchmarking Instruction Sensitivity for Large Audio Language Models
Abstract
Large Audio Language Models (LALMs), which couple acoustic perception with large language models (LLMs) to extract and understand diverse information from audio, have attracted intense interest from both academic and industrial communities. However, existing LALMs are highly sensitive to how instructions are phrased, affecting both (i) instruction-following rates and (ii) task performance. Yet, no existing benchmarks offer a systematic and comprehensive evaluation of this sensitivity. We introduce ISA-Bench, a dynamic benchmark evaluating instruction sensitivity for LALMs along three axes: instruction description, output format, and task composition. We assess recent open-source and proprietary LALMs using ISA-Bench, profiling both compliance and accuracy under controlled instruction variations. Experimental results reveal that even state-of-the-art LALMs suffer significant instruction sensitivity, leading to degraded performance on fundamental audio understanding tasks. To mitigate this issue, we fine-tune Qwen2-Audio on a specifically constructed complex instruction-variant dataset, achieving a marked improvement in instruction-following performance. However, this also induces nontrivial catastrophic forgetting: the model loses some previously mastered task capabilities when exposed to new instruction styles. Our benchmark provides a standardized basis for assessing and improving instruction sensitivity in LALMs, underscoring the need for instruction-robust audio understanding in real-world pipelines.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of ISA-Bench, a comprehensive benchmark for evaluating instruction sensitivity in large audio language models, which reveals significant challenges and trade-offs in model performance. The paper's technical contributions, innovative methodology, and rigorous experimental evaluation position it as a valuable resource for advancing research in audio language processing.
Methodology
The paper introduces ISA-Bench, a novel benchmark designed to evaluate instruction sensitivity in large audio language models (LALMs) across three dimensions: instruction description, output format, and task composition. This multidimensional approach is a significant improvement over existing benchmarks, which typically focus on only one aspect of instruction sensitivity. The methodology is well-structured, employing a systematic way to generate instruction variants and assess model performance through compliance-aware metrics. The use of diverse instruction variants generated by LLMs adds depth to the evaluation process, ensuring that the benchmark reflects real-world scenarios.
Results & evaluation
The experimental section is robust, featuring a comprehensive evaluation of several state-of-the-art LALMs across five atomic tasks. The results highlight the significant challenges posed by instruction sensitivity, with clear evidence that even top-performing models struggle under varied instruction forms. The fine-tuning experiments conducted on Qwen2-Audio provide valuable insights into the trade-offs between improving instruction-following ability and the risk of catastrophic forgetting, demonstrating a thorough understanding of the complexities involved in training LALMs.
Reproducibility
The paper provides a clear description of the experimental setup, including the models tested and the datasets used. However, specific implementation details, such as hyperparameters for fine-tuning and the exact nature of the instruction variants, are somewhat lacking. While the GitHub repository offers a project URL, which may contain additional resources, the reproducibility of results could be enhanced with more detailed methodological transparency.
Limitations
One notable limitation is the potential for catastrophic forgetting observed during fine-tuning, which could hinder the practical deployment of LALMs in real-world applications. Additionally, the benchmark may not fully capture all nuances of instruction sensitivity, particularly in more complex or less common audio tasks. The reliance on specific datasets may also limit the generalizability of the findings.
Broader impact
The development of ISA-Bench has significant implications for the field of audio language processing, as it addresses a critical gap in the evaluation of LALMs. By highlighting the importance of instruction robustness, the benchmark encourages future research to focus on improving model performance in real-world scenarios, ultimately enhancing the reliability of human-computer interaction through audio understanding. The main contribution of this paper is the introduction of ISA-Bench, a comprehensive benchmark for evaluating instruction sensitivity in large audio language models, which reveals significant challenges and trade-offs in model performance. The paper's technical contributions, innovative methodology, and rigorous experimental evaluation position it as a valuable resource for advancing research in audio language processing.
Author of this paper? Send a correction.