MuQ-Eval: An Open-Source Per-Sample Quality Metric for AI Music Generation Evaluation
Abstract
Distributional metrics such as Fréchet Audio Distance cannot score individual music clips and correlate poorly with human judgments, while the only per-sample learned metric achieving high human correlation is closed-source. We introduce MUQ-EVAL, an open-source per-sample quality metric for AIgenerated music built by training lightweight prediction heads on frozen MuQ-310M features using MusicEval, a dataset of generated clips from 31 text-to-music systems with expert quality ratings. Our simplest model, frozen features with attention pooling and a two-layer MLP, achieves system-level SRCC = 0.957 and utterance-level SRCC = 0.838 with human mean opinion scores. A systematic ablation over training objectives and adaptation strategies shows that no addition meaningfully improves the frozen baseline, indicating that frozen MuQ representations already capture quality-relevant information. Encoder choice is the dominant design factor, outweighing all architectural and training decisions. LoRA-adapted models trained on as few as 150 clips already achieve usable correlation, enabling personalized quality evaluators from individual listener annotations. A controlled degradation analysis reveals selective sensitivity to signal-level artifacts but insensitivity to musical-structural distortions. Our metric, MUQ-EVAL, is fully open-source, outperforms existing open per-sample metrics, and runs in real time on a single consumer GPU. Code, model weights, and evaluation scripts are available at https://github.com/dgtql/MuQ-Eval.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of MUQ-EVAL, an open-source per-sample quality metric for AI-generated music that achieves high correlation with human ratings, demonstrating the effectiveness of frozen representations from a music understanding encoder. This work is significant as it addresses a critical gap in the evaluation of AI-generated music, providing a practical and accessible tool for researchers and practitioners in the field.
Methodology
The paper presents a well-structured methodology for developing a per-sample quality metric for AI-generated music, leveraging frozen MuQ-310M features and a simple MLP architecture. The systematic ablation study is a strong point, clearly demonstrating that the frozen features alone are sufficient for achieving high correlation with human ratings, thus emphasizing the importance of encoder choice over complex training strategies. The approach is innovative in its use of lightweight models and attention pooling, making it accessible for individual researchers.
Results & evaluation
The experimental setup is robust, utilizing a comprehensive dataset (MusicEval) with expert ratings across multiple text-to-music systems. The results show high system-level and utterance-level correlations with human mean opinion scores, indicating the effectiveness of the proposed metric. The controlled degradation analysis adds depth to the evaluation, although the selective sensitivity to signal-level artifacts versus musical-structural distortions raises questions about the metric's applicability in all contexts.
Reproducibility
The authors provide clear implementation details, including training configurations and computational requirements, which enhance reproducibility. The open-source nature of the project, with code and model weights available on GitHub, further supports this aspect, allowing other researchers to replicate the findings easily.
Limitations
The paper acknowledges limitations, particularly in the metric's insensitivity to musical-structural distortions and the potential generalization issues across different datasets. The reliance on a single dataset for evaluation may limit the broader applicability of the findings, and the authors suggest future work to address these concerns.
Broader impact
The proposed metric has significant implications for the music generation community, providing a practical tool for quality assessment that can facilitate the development and comparison of generative models. The ability to create personalized quality evaluators from small annotation budgets could democratize access to quality assessment tools, enabling more creators to engage with AI music generation. The main contribution of this paper is the introduction of MUQ-EVAL, an open-source per-sample quality metric for AI-generated music that achieves high correlation with human ratings, demonstrating the effectiveness of frozen representations from a music understanding encoder. This work is significant as it addresses a critical gap in the evaluation of AI-generated music, providing a practical and accessible tool for researchers and practitioners in the field.
Author of this paper? Send a correction.