MAEB: Massive Audio Embedding Benchmark
Abstract
We introduce the Massive Audio Embedding Benchmark (MAEB), a large-scale benchmark covering 30 tasks across speech, music, environmental sounds, and cross-modal audio-text reasoning in 100+ languages. We evaluate 50+ models and find that no single model dominates across all tasks: contrastive audio-text models excel at environmental sound classification (e.g., ESC50) but score near random on multilingual speech tasks (e.g., SIB-FLEURS), while speech-pretrained models show the opposite pattern. Clustering remains challenging for all models, with even the best-performing model achieving only modest results. We observe that models excelling on acoustic understanding often perform poorly on linguistic tasks, and vice versa. We also show that the performance of audio encoders on MAEB correlates highly with their performance when used in audio large language models. MAEB is derived from MAEB+, a collection of 98 tasks. MAEB is designed to maintain task diversity while reducing evaluation cost, and it integrates into the MTEB ecosystem for unified evaluation across text, image, and audio modalities. We release MAEB and all 98 tasks along with code and a leaderboard at https://github.com/embeddings-benchmark/mteb.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces the Massive Audio Embedding Benchmark (MAEB), a significant contribution to the field of audio machine learning that provides a comprehensive evaluation framework across diverse tasks and languages. The methodology and experimental results offer valuable insights into model performance, although further statistical analysis and detailed reproducibility guidelines would enhance its impact.
Methodology
The methodology presented in the paper is robust, introducing a comprehensive benchmark (MAEB) that spans multiple audio tasks and languages. The authors provide a clear rationale for the selection of tasks and models, and the integration into the MTEB ecosystem is a significant step towards unified evaluation across modalities. However, the paper could benefit from a more detailed description of the benchmarking process and the specific metrics used for evaluation.
Results & evaluation
The experiments are extensive, evaluating over 50 models across 30 tasks. The results highlight the performance discrepancies between models trained for different audio tasks, which is a valuable insight for future research. However, the paper lacks a thorough statistical analysis of the results, which would strengthen the claims made regarding model performance.
Reproducibility
The authors have committed to releasing code and a leaderboard, which is commendable and supports reproducibility. However, the paper should include more detailed instructions on how to replicate the experiments, including specific configurations and hyperparameters used for each model.
Limitations
One limitation noted is the performance of models on clustering tasks, where even the best-performing model achieves only modest results. Additionally, the paper acknowledges the trade-offs between acoustic understanding and linguistic tasks, which may limit the applicability of certain models across all tasks.
Broader impact
The MAEB benchmark has the potential to significantly impact the field of audio machine learning by providing a standardized evaluation framework. This could lead to improved model development and encourage further research into multilingual and cross-modal audio tasks. The release of the benchmark also promotes collaboration and transparency in the research community. The paper introduces the Massive Audio Embedding Benchmark (MAEB), a significant contribution to the field of audio machine learning that provides a comprehensive evaluation framework across diverse tasks and languages. The methodology and experimental results offer valuable insights into model performance, although further statistical analysis and detailed reproducibility guidelines would enhance its impact.
Author of this paper? Send a correction.