FUSEMOS: Perceptual Evaluation of Text-to-Music Generation with Dual-Encoder Fusion and Ranking-Aware Composite Loss
Abstract
Perceptual evaluation of generated music is essential for developing text-to-music (TTM) generation systems. Existing automatic music perceptual evaluation methods rely solely on the contrastive language audio pretraining (CLAP) model to extract both audio and text features for predicting human-annotated mean opinion scores (MOS), particularly for overall musical impression (OMI) and textual alignment (TA). However, CLAP is limited in capturing fine-grained musical attributes such as timbre and expressiveness \emph{etc}. To address this, we propose FUSEMOS, a dual-encoder fusion framework that integrates CLAP with MERT, a self-supervised model pretrained on large scale music audio, to better capture both high level semantic alignment and detailed musical characteristics for more accurate OMI and TA estimation. Furthermore, to overcome the limitations of conventional regression losses in measuring human preferred relative ranking of music segments, we design a novel ranking-aware composite loss function. Experiments conducted on the MusicEval benchmark demonstrate that our approach outperforms the existing SOTA methods in both mean squared error and rank correlation metrics by a large margin.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
FUSEMOS proposes a dual-encoder fusion framework combining CLAP and MERT with a ranking-aware composite loss to improve the accuracy of automatic perceptual evaluation for text-to-music generation systems. The paper demonstrates significant improvements in rank correlation metrics over single-encoder baselines on the MusicEval benchmark, validating the complementary nature of semantic and structural audio representations for MOS prediction.
Methodology
The paper proposes FUSEMOS, a dual-encoder framework that fuses features from CLAP (for semantic alignment) and MERT (for musical structure) to predict MOS scores for Overall Musical Impression (OMI) and Textual Alignment (TA). The core architectural contribution is a "late fusion" strategy where independent MLP heads predict scores from each encoder, which are then combined via a linear layer. This is a standard ensemble technique rather than a deep feature-level fusion. The loss function combines a truncated regression loss (Huber-like) with a contrastive ranking loss. While the motivation to capture ordinal preferences is sound, the specific formulation of the ranking loss is a minor adaptation of existing contrastive learning objectives. The methodology is competent but relies heavily on existing pretrained models without significant architectural innovation in the fusion mechanism itself.
Results & evaluation
Experiments are conducted on the MusicEval benchmark, which is a relevant and recent dataset for TTM evaluation. The results show substantial improvements over the CLAP-only baseline, particularly in rank correlation metrics (SRCC, KTAU). The ablation studies are thorough, validating the necessity of both encoders, the choice of late fusion over early fusion/attention, and the components of the composite loss. The improvement in S-SRCC from 0.861 to 0.940 for TA is significant. However, the comparison is limited to the baseline provided by the dataset creators; comparisons with other recent MOS prediction methods (e.g., UTMOS variants adapted for music, or other dual-encoder approaches) are absent.
Reproducibility
The paper provides specific hyperparameters (learning rate, batch size, margins) and references specific pretrained checkpoints (HuggingFace links for CLAP and MERT). The use of a single H800 GPU and SGD optimizer is noted. However, the code repository is not provided in the text, and the specific implementation details of the "learnable query-based attention pooling" in the MERT branch are described briefly. The reliance on a specific, potentially proprietary or limited-access dataset (AISHELL_7A) may hinder full reproduction by the broader community if the dataset is not widely available.
Limitations
The primary limitation is the lack of generalization testing. The model is trained and tested on the MusicEval benchmark, which consists of clips from 31 specific models. It is unclear how well FUSEMOS generalizes to music generated by unseen models or different TTM architectures. Additionally, the "late fusion" approach is computationally redundant as it requires running two large encoders and multiple MLP heads, which could be a bottleneck for real-time evaluation. The paper does not discuss the computational cost or latency of the inference process.
Broader impact
This work contributes to the development of objective metrics for text-to-music generation, which is a rapidly growing field. By improving the correlation between automatic scores and human perception, it enables more efficient iteration and benchmarking of TTM systems. The integration of musical structure models (MERT) with semantic alignment models (CLAP) sets a precedent for multimodal evaluation frameworks. However, the impact is currently limited to the specific domain of TTM evaluation and does not extend to broader audio quality assessment tasks. FUSEMOS proposes a dual-encoder fusion framework combining CLAP and MERT with a ranking-aware composite loss to improve the accuracy of automatic perceptual evaluation for text-to-music generation systems. The paper demonstrates significant improvements in rank correlation metrics over single-encoder baselines on the MusicEval benchmark, validating the complementary nature of semantic and structural audio representations for MOS prediction.
Author of this paper? Send a correction.