AudioEval: Automatic Dual-Perspective and Multi-Dimensional Evaluation of Text-to-Audio-Generation
Abstract
Text-to-audio (TTA) is rapidly advancing, with broad potential in virtual reality, accessibility, and creative media. However, evaluating TTA quality remains difficult: human ratings are costly and limited, while existing objective metrics capture only partial aspects of perceptual quality. To address this gap, we introduce AudioEval, the first large-scale TTA evaluation dataset, containing 4,200 audio samples from 24 systems with 126,000 ratings across five perceptual dimensions, annotated by both experts and non-experts. Based on this resource, we propose Qwen-DisQA, a multimodal scoring model that jointly processes text prompts and generated audio to predict human-like quality ratings. Experiments show its effectiveness in providing reliable and scalable evaluation. The dataset will be made publicly available to accelerate future research.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of AudioEval, the first large-scale dataset for evaluating text-to-audio generation, along with the Qwen-DisQA model that predicts human-like quality ratings across multiple dimensions. This work represents a substantial step forward in addressing the challenges of evaluating TTA systems, with the potential to influence future research and applications in the field.
Methodology
The methodology presented in the paper is robust, introducing the AudioEval dataset as a pioneering resource for TTA evaluation, which is a significant advancement in the field. The dual-perspective annotation by both experts and non-experts enhances the dataset's reliability and applicability. The Qwen-DisQA model effectively integrates multimodal inputs, addressing the complexities of audio generation and evaluation. The approach of predicting a distribution of ratings rather than a single score is innovative, allowing for a more nuanced understanding of quality.
Results & evaluation
The experiments are well-structured, with a clear delineation of training, validation, and testing protocols. The comparative analysis against existing models demonstrates the effectiveness of Qwen-DisQA, showing significant improvements in correlation and reliability. The use of multiple evaluation metrics, including MSE and PCC, provides a comprehensive assessment of model performance. However, the paper could benefit from a more detailed discussion of the specific datasets used for comparison.
Reproducibility
While the paper outlines the training configuration and evaluation metrics, it lacks detailed implementation specifics such as hyperparameter settings, code availability, or links to the dataset. This omission could hinder reproducibility for other researchers looking to validate or build upon the findings.
Limitations
One limitation is the reliance on subjective ratings, which, despite being mitigated by dual perspectives, can still introduce variability. Additionally, the dataset's size and diversity, while substantial, may not encompass all possible audio generation scenarios, potentially limiting the model's generalizability. The paper does not address the potential biases in expert versus non-expert ratings.
Broader impact
The work has significant implications for various fields, including virtual reality, accessibility, and creative media, where TTA systems are increasingly relevant. By providing a reliable evaluation framework, the research can accelerate advancements in TTA technology, fostering innovation and enhancing user experiences across applications. The main contribution of this paper is the introduction of AudioEval, the first large-scale dataset for evaluating text-to-audio generation, along with the Qwen-DisQA model that predicts human-like quality ratings across multiple dimensions. This work represents a substantial step forward in addressing the challenges of evaluating TTA systems, with the potential to influence future research and applications in the field.
Author of this paper? Send a correction.