AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answering
Abstract
Although text-to-audio generation has made remarkable progress in realism and diversity, the development of evaluation metrics has not kept pace. Widely-adopted approaches, typically based on embedding similarity like CLAPScore, effectively measure general relevance but remain limited in fine-grained semantic alignment and compositional reasoning. To address this, we introduce AQAScore, a backbone-agnostic evaluation framework that leverages the reasoning capabilities of audio-aware large language models (ALLMs). AQAScore reformulates assessment as a probabilistic semantic verification task; rather than relying on open-ended text generation, it estimates alignment by computing the exact log-probability of a "Yes" answer to targeted semantic queries. We evaluate AQAScore across multiple benchmarks, including human-rated relevance, pairwise comparison, and compositional reasoning tasks. Experimental results show that AQAScore consistently achieves higher correlation with human judgments than similarity-based metrics and generative prompting baselines, showing its effectiveness in capturing subtle semantic inconsistencies and scaling with the capability of underlying ALLMs.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of AQAScore, a novel evaluation framework that enhances the assessment of semantic alignment in text-to-audio generation. The comprehensive analysis highlights the innovative methodology, robust experimental validation, and the potential for broader applications in the field, marking a significant advancement in evaluation metrics for audio generation tasks.
Methodology
The AQAScore framework introduces a novel approach to evaluating text-to-audio generation by reformulating the evaluation as a probabilistic semantic verification task. This methodology leverages audio-aware large language models (ALLMs) to assess semantic alignment through targeted queries, which is a significant shift from traditional embedding similarity metrics. The backbone-agnostic nature of AQAScore enhances its applicability across different models, making it a versatile tool for researchers in the field.
Results & evaluation
The experimental setup is robust, involving multiple benchmarks that include human-rated relevance and pairwise comparisons. The results demonstrate a consistent improvement in correlation with human judgments compared to existing metrics, indicating that AQAScore effectively captures subtle semantic inconsistencies. The thorough evaluation across diverse tasks adds credibility to the findings and showcases the framework's strengths.
Reproducibility
While the paper outlines the methodology and experimental setups, it lacks specific implementation details that would facilitate reproducibility. The absence of a publicly available code repository or demo limits the ability for other researchers to validate the findings independently.
Limitations
The paper acknowledges limitations, such as the potential biases in human ratings and the reliance on the capabilities of underlying ALLMs. Additionally, the framework's performance may vary depending on the specific models used, which could affect generalizability.
Broader impact
AQAScore has the potential to significantly influence the evaluation metrics used in text-to-audio generation, paving the way for more nuanced assessments of semantic alignment. This could lead to advancements in applications such as audio generation for multimedia content, enhancing user experiences in various domains. The main contribution of this paper is the introduction of AQAScore, a novel evaluation framework that enhances the assessment of semantic alignment in text-to-audio generation. The comprehensive analysis highlights the innovative methodology, robust experimental validation, and the potential for broader applications in the field, marking a significant advancement in evaluation metrics for audio generation tasks.
Author of this paper? Send a correction.