Rubric-Based Optimization for Text-to-Music Generation
Abstract
Post-training text-to-music generation requires reward signals that capture multiple aspects of musical quality beyond what any single automatic metric can measure. We study structured, rubric-based rewards from pretrained audio-language models (ALMs) as training signals for both autoregressive and diffusion-based music generators. An ALM scores each generated clip against the rubric; we rank candidates generated for the same text prompt by their scores and convert these rankings into preference pairs for DPO on both MusicGen-small and ACE-Step~v1, and additionally use the rubric scores directly as scalar rewards for DiffusionNFT on ACE-Step~v1. On MusicCaps, rubric-based optimization improves CLAP, SongEval, and Audiobox-Aesthetics simultaneously, with the strongest gains obtained by DiffusionNFT on ACE-Step. By contrast, on MusicGen-small, building preferences from any one of these automatic evaluators produces clear cross-metric trade-offs: the targeted evaluator improves while other independent evaluators deteriorate. We further study tempo, key, and instrumentation, where precise objective rewards are available. Directly optimizing these specialized rewards reliably improves the target attributes, whereas ALM rubrics provide only partial transfer for tempo and instrumentation and no measurable improvement for key. Together, these results suggest a practical division of labor: ALM rubrics are effective for broad perceptual qualities that are difficult to formalize, while specialized objective rewards remain preferable when reliable measurements are available.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a rubric-based optimization framework using off-the-shelf audio-language models to improve text-to-music generation, demonstrating that structured, multi-dimensional rewards from ALMs can simultaneously enhance multiple quality metrics without the trade-offs associated with single-metric optimization, while establishing a clear division of labor between rubric rewards for perceptual qualities and objective rewards for measurable attributes.
Methodology
The paper proposes a structured, rubric-based reward system using off-the-shelf Audio-Language Models (ALMs) to post-train text-to-music generators. The methodology is sound, leveraging two distinct optimizers: DPO for autoregressive models (MusicGen) and DiffusionNFT for diffusion models (ACE-Step). A key strength is the rigorous comparison between "joint" and "dimension-wise" prompting strategies, revealing that independent evaluation of rubric dimensions reduces bias and improves signal quality. The inclusion of a "division of labor" analysis—comparing rubric rewards against precise objective rewards for attributes like tempo and key—is methodologically sophisticated and provides actionable insights for practitioners.
Results & evaluation
The experiments are comprehensive, covering two generator architectures, three ALM raters, and two prompting strategies. The results clearly demonstrate that rubric-based rewards avoid the "reward hacking" and cross-metric trade-offs observed when optimizing single automatic metrics (like CLAP or SongEval) in isolation. The human listening test, while small (n=7), provides crucial validation that the automatic metric improvements translate to perceptual quality. The ablation on objective attributes (tempo/key) effectively highlights the limitations of ALMs for precise, measurable tasks, reinforcing the paper's central thesis.
Reproducibility
The paper offers high reproducibility standards for a preprint. It provides detailed hyperparameters, specific model checkpoints, and the exact rubric schema with prompt versions and hashes. The inclusion of the full JSON parsing logic and reward calculation formulas allows for faithful re-implementation. The use of open-weight models for both generators and raters further lowers the barrier to entry for replication.
Limitations
The primary limitation is the reliance on open-weight ALMs, which may be less capable than proprietary closed models. The human evaluation sample size is small, limiting statistical power. Additionally, the study is limited to short clips (30s), which may not capture long-form musical structure. The performance of the rubric approach is highly sensitive to the specific ALM used, with some configurations (e.g., Music Flamingo with joint prompting) leading to performance degradation.
Broader impact
This work provides a practical framework for aligning generative audio models with human preferences without the cost of large-scale human annotation. It highlights the potential of LLMs/ALMs as "judges" for fine-grained audio quality control. The findings on the complementary nature of rubric and objective rewards will likely influence future hybrid reward systems in the field of generative audio. The paper introduces a rubric-based optimization framework using off-the-shelf audio-language models to improve text-to-music generation, demonstrating that structured, multi-dimensional rewards from ALMs can simultaneously enhance multiple quality metrics without the trade-offs associated with single-metric optimization, while establishing a clear division of labor between rubric rewards for perceptual qualities and objective rewards for measurable attributes.
Author of this paper? Send a correction.