UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Models
Abstract
The development of audio foundation models has accelerated rapidly since the emergence of GPT-4o. However, the lack of comprehensive evaluation has become a critical bottleneck for further progress in the field, particularly in audio generation. Current audio evaluation faces three major challenges: (1) audio evaluation lacks a unified framework, with datasets and code scattered across various sources, hindering fair and efficient cross-model comparison;(2) audio codecs, as a key component of audio foundation models, lack a widely accepted and holistic evaluation methodology; (3) existing speech benchmarks are heavily reliant on English, making it challenging to objectively assess models' performance on Chinese. To address the first issue, we introduce UltraEval-Audio, a unified evaluation framework for audio foundation models, specifically designed for both audio understanding and generation tasks. UltraEval-Audio features a modular architecture, supporting 10 languages and 14 core task categories, while seamlessly integrating 24 mainstream models and 36 authoritative benchmarks. To enhance research efficiency, the framework provides a one-command evaluation feature, accompanied by real-time public leaderboards. For the second challenge, UltraEval-Audio adopts a novel comprehensive evaluation scheme for audio codecs, evaluating performance across three key dimensions: semantic accuracy, timbre fidelity, and acoustic quality. To address the third issue, we propose two new Chinese benchmarks, SpeechCMMLU and SpeechHSK, designed to assess Chinese knowledge proficiency and language fluency. We wish that UltraEval-Audio will provide both academia and industry with a transparent, efficient, and fair platform for comparison of audio models. Our code, benchmarks, and leaderboards are available at https://github.com/OpenBMB/UltraEval-Audio.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of UltraEval-Audio, a unified framework for evaluating audio foundation models that addresses critical evaluation challenges and enhances research efficiency. This work is significant as it not only fills existing gaps in the evaluation landscape but also sets a foundation for future advancements in audio processing and understanding.
Methodology
The proposed methodology introduces a comprehensive and modular evaluation framework that addresses critical gaps in the evaluation of audio foundation models. It effectively integrates multiple languages and tasks while providing a systematic approach to codec evaluation. The framework's design allows for easy adaptation and extensibility, which is crucial for future research.
Results & evaluation
The experiments conducted demonstrate the framework's capability to evaluate a diverse set of audio models and codecs across multiple benchmarks. The inclusion of new Chinese benchmarks adds significant value, addressing a notable gap in existing evaluations. The results are presented clearly, allowing for meaningful comparisons across models.
Reproducibility
The paper provides a clear description of the framework's architecture and evaluation processes, along with a publicly accessible code repository. This enhances reproducibility, as other researchers can replicate the evaluation setup and results. However, some specifics about the experimental setup could be elaborated for complete clarity.
Limitations
While the framework is comprehensive, it may still face challenges in evaluating models that operate in less common languages or dialects. Additionally, the reliance on existing benchmarks may limit the novelty of some evaluation metrics.
Broader impact
The UltraEval-Audio framework has the potential to significantly advance the field of audio foundation models by providing a standardized evaluation methodology. This can lead to improved model development and performance assessment, particularly in multilingual contexts. The main contribution of this paper is the introduction of UltraEval-Audio, a unified framework for evaluating audio foundation models that addresses critical evaluation challenges and enhances research efficiency. This work is significant as it not only fills existing gaps in the evaluation landscape but also sets a foundation for future advancements in audio processing and understanding.
Author of this paper? Send a correction.