CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction
Abstract
While music generation models have evolved to handle complex multimodal inputs mixing text, lyrics, and reference audio, evaluation mechanisms have lagged behind. In this paper, we bridge this critical gap by establishing a comprehensive ecosystem for music reward modeling under Compositional Multimodal Instruction (CMI), where the generated music may be conditioned on text descriptions, lyrics, and audio prompts. We first introduce CMI-Pref-Pseudo, a large-scale preference dataset comprising 110k pseudo-labeled samples, and CMI-Pref, a high-quality, human-annotated corpus tailored for fine-grained alignment tasks. To unify the evaluation landscape, we propose CMI-RewardBench, a unified benchmark that evaluates music reward models on heterogeneous samples across musicality, text-music alignment, and compositional instruction alignment. Leveraging these resources, we develop CMI reward models (CMI-RMs), a parameter-efficient reward model family capable of processing heterogeneous inputs. We evaluate their correlation with human judgments scores on musicality and alignment on CMI-Pref along with previous datasets. Further experiments demonstrate that CMI-RM not only correlates strongly with human judgments, but also enables effective inference-time scaling via top-k filtering. The necessary training data, benchmarks, and reward models are publicly available.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a comprehensive framework for evaluating music generation models through a novel benchmark and datasets, significantly advancing the state of the art in music reward modeling. The methodology is rigorous, and the results demonstrate a clear improvement in aligning model outputs with human preferences, making it a valuable contribution to the field of machine learning and music technology.
Methodology
The paper introduces a novel framework for evaluating music generation models using Compositional Multimodal Instruction (CMI). It constructs two datasets—CMI-Pref-Pseudo and CMI-Pref—alongside a unified benchmark, CMI-RewardBench, which assesses models on multiple dimensions of musicality and alignment. The methodology is robust, utilizing both pseudo-labeling and expert annotations, and employs a parameter-efficient architecture for the reward models, allowing for effective processing of heterogeneous inputs. The two-stage training strategy enhances the model's performance by leveraging both large-scale pseudo-labeled data and high-quality human annotations.
Results & evaluation
The experiments are comprehensive, demonstrating the effectiveness of the proposed CMI-RM against existing baselines across various tasks. The results show strong correlations with human judgments, indicating that the proposed models can effectively evaluate music generation quality. The paper provides detailed metrics and comparisons, showcasing the advantages of the CMI-RewardBench in capturing the nuances of human preferences in music generation.
Reproducibility
The authors have made their datasets, benchmark, and model weights publicly available, which enhances reproducibility. The detailed methodology, including the training protocols and evaluation metrics, is well-documented, allowing other researchers to replicate the experiments. However, the reliance on specific models for pseudo-labeling may introduce variability that is not fully accounted for.
Limitations
One limitation is the potential bias in the pseudo-labeling process, which may affect the quality of the training data. Additionally, while the framework addresses the complexity of multimodal inputs, the evaluation may still be subjective, as musicality and alignment can vary significantly based on individual listener preferences. The paper also does not extensively discuss the scalability of the approach to larger datasets or different musical genres.
Broader impact
This work has significant implications for the field of music generation and evaluation, providing a structured approach to assess models that can handle complex multimodal inputs. The availability of the datasets and benchmark can spur further research in aligned music generation and improve the quality of AI-generated music in commercial applications. The methodology could also be adapted for use in other creative domains where multimodal inputs are prevalent. This paper presents a comprehensive framework for evaluating music generation models through a novel benchmark and datasets, significantly advancing the state of the art in music reward modeling. The methodology is rigorous, and the results demonstrate a clear improvement in aligning model outputs with human preferences, making it a valuable contribution to the field of machine learning and music technology.
Author of this paper? Send a correction.