BRACE: A Benchmark for Robust Audio Caption Quality Evaluation
Abstract
Automatic audio captioning is essential for audio understanding, enabling applications such as accessibility and content indexing. However, evaluating the quality of audio captions remains a major challenge, especially in reference-free settings where high-quality ground-truth captions are unavailable. While CLAPScore is currently the most widely used reference-free Audio Caption Evaluation Metric(ACEM), its robustness under diverse conditions has not been systematically validated. To address this gap, we introduce BRACE, a new benchmark designed to evaluate audio caption alignment quality in a reference-free setting. BRACE is primarily designed for assessing ACEMs, and can also be extended to measure the modality alignment abilities of Large Audio Language Model(LALM). BRACE consists of two sub-benchmarks: BRACE-Main for fine-grained caption comparison and BRACE-Hallucination for detecting subtle hallucinated content. We construct these datasets through high-quality filtering, LLM-based corruption, and human annotation. Given the widespread adoption of CLAPScore as a reference-free ACEM and the increasing application of LALMs in audio-language tasks, we evaluate both approaches using the BRACE benchmark, testing CLAPScore across various CLAP model variants and assessing multiple LALMs. Notably, even the best-performing CLAP-based ACEM achieves only a 70.01 F1-score on the BRACE-Main benchmark, while the best LALM reaches just 63.19. By revealing the limitations of CLAP models and LALMs, our BRACE benchmark offers valuable insights into the direction of future research.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of BRACE, a benchmark for evaluating audio caption quality in a reference-free setting, which addresses critical gaps in the assessment of audio captioning metrics. This work significantly advances the field by providing a structured approach to evaluate model performance and identify areas for improvement, thereby fostering future research in audio-language understanding.
Methodology
The paper introduces BRACE, a benchmark specifically designed for evaluating audio captioning metrics in a reference-free setting. It comprises two sub-benchmarks, BRACE-Main and BRACE-Hallucination, which assess fine-grained caption comparisons and hallucination detection, respectively. The methodology is robust, utilizing a combination of high-quality filtering, LLM-based corruption, and human annotation to construct datasets. The dual focus on both the quality of audio-caption alignment and the detection of hallucinations presents a comprehensive approach to addressing existing gaps in audio caption evaluation metrics. The use of diverse models and evaluation strategies enhances the credibility of the findings.
Results & evaluation
The experiments conducted on BRACE benchmark reveal significant insights into the performance of CLAP-based ACEMs and LALMs. The results indicate that even the best-performing models struggle to achieve high scores, highlighting the challenges in audio caption evaluation. The evaluation metrics are well-defined, and the performance of various models is systematically compared, providing a clear understanding of their limitations. The rigorous testing across different model architectures adds depth to the experimental evaluation.
Reproducibility
The authors have taken steps to ensure reproducibility by providing access to the evaluation code and benchmark datasets. Detailed descriptions of the experimental configurations, including model settings and evaluation strategies, are included. However, the paper could benefit from more explicit instructions on how to replicate the experiments, particularly regarding the specific prompts and configurations used in LALM evaluations.
Limitations
The paper acknowledges certain limitations, particularly regarding the performance of existing models on the benchmark. However, it could further elaborate on potential biases in the dataset construction process and the implications of using LLMs for generating and corrupting captions. Additionally, the computational constraints faced during experiments limit the ability to conduct extensive evaluations, which could affect the generalizability of the results.
Broader impact
The development of BRACE has significant implications for the field of audio understanding, particularly in enhancing accessibility and content indexing. By providing a reliable benchmark for evaluating audio captioning metrics, it can drive improvements in model development and evaluation practices. However, the potential for misuse of audio captioning technologies, such as generating misleading or inaccurate captions, should be considered, and appropriate safeguards should be discussed. The main contribution of this paper is the introduction of BRACE, a benchmark for evaluating audio caption quality in a reference-free setting, which addresses critical gaps in the assessment of audio captioning metrics. This work significantly advances the field by providing a structured approach to evaluate model performance and identify areas for improvement, thereby fostering future research in audio-language understanding.
Author of this paper? Send a correction.