AURA Score: A Metric For Holistic Audio Question Answering Evaluation
Abstract
Audio Question Answering (AQA) is a key task for evaluating Audio-Language Models (ALMs), yet assessing open-ended responses remains challenging. Existing metrics used for AQA such as BLEU, METEOR and BERTScore, mostly adapted from NLP and audio captioning, rely on surface similarity and fail to account for question context, reasoning, and partial correctness. To address the gap in literature, we make three contributions in this work. First, we introduce AQEval to enable systematic benchmarking of AQA metrics. It is the first benchmark of its kind, consisting of 10k model responses annotated by multiple humans for their correctness and relevance. Second, we conduct a comprehensive analysis of existing AQA metrics on AQEval, highlighting weak correlation with human judgment, especially for longer answers. Third, we propose a new metric - AURA score, to better evaluate open-ended model responses. On AQEval, AURA achieves state-of-the-art correlation with human ratings, significantly outperforming all baselines. Through this work, we aim to highlight the limitations of current AQA evaluation methods and motivate better metrics. We release both the AQEval benchmark and the AURA metric to support future research in holistic AQA evaluation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of AQEval and the AURA score, which together provide a comprehensive framework for evaluating open-ended responses in audio question answering. This work addresses critical shortcomings in existing evaluation metrics and sets a new standard for future research in the field.
Methodology
The paper introduces AQEval, a novel benchmark for Audio Question Answering (AQA) metrics, which is a significant advancement in evaluating open-ended responses in audio contexts. The methodology employs a combination of human annotations and a new metric, AURA, which integrates reasoning capabilities of large language models (LLMs) with an audio entailment component. This dual approach allows for a more nuanced evaluation of responses, addressing the limitations of existing metrics that primarily focus on surface-level similarity.
Results & evaluation
The experimental setup is robust, utilizing a dataset of 10k annotated responses that allows for systematic benchmarking of AQA metrics. The authors provide a comprehensive analysis of existing metrics, demonstrating their weak correlation with human judgments, particularly for longer answers. AURA is shown to outperform traditional metrics significantly, achieving state-of-the-art correlation with human ratings. The ablation studies further validate the effectiveness of the proposed methodology.
Reproducibility
The paper includes detailed descriptions of the dataset construction, annotation process, and experimental setup, which enhances reproducibility. However, the reliance on specific LLMs for scoring may limit the generalizability of the results to other models or contexts.
Limitations
While the paper addresses significant gaps in AQA evaluation, it does not explore the potential biases in human annotations or the limitations of the LLMs used. Additionally, the performance of AURA in real-world applications remains to be fully validated.
Broader impact
The introduction of AQEval and AURA has the potential to significantly influence future research in audio-language models and their evaluation. By providing a more accurate assessment of model responses, this work can lead to improvements in the development of ALMs and their applications in various domains, including accessibility, education, and content creation. The main contribution of this paper is the introduction of AQEval and the AURA score, which together provide a comprehensive framework for evaluating open-ended responses in audio question answering. This work addresses critical shortcomings in existing evaluation metrics and sets a new standard for future research in the field.
Author of this paper? Send a correction.