UrgentMOS: Unified Multi-Metric and Preference Learning for Robust Speech Quality Assessment
Abstract
Automatic speech quality assessment has become increasingly important as modern speech generation systems continue to advance, while human listening tests remain costly, time-consuming, and difficult to scale. Most existing learning-based assessment models rely primarily on scarce human-annotated mean opinion score (MOS) data, which limits robustness and generalization, especially when training across heterogeneous datasets. In this work, we propose UrgentMOS, a unified speech quality assessment framework that jointly learns from diverse objective and perceptual quality metrics, while explicitly tolerating the absence of arbitrary subsets of metrics during training. By leveraging complementary quality facets under heterogeneous supervision, UrgentMOS enables effective utilization of partially annotated data and improves robustness when trained on large-scale, multi-source datasets. Beyond absolute score prediction, UrgentMOS explicitly models pairwise quality preferences by directly predicting comparative MOS (CMOS), making it well suited for preference-based evaluation scenarios commonly adopted in system benchmarking. Extensive experiments across a wide range of speech quality datasets, including simulated distortions, speech enhancement, and speech synthesis, demonstrate that UrgentMOS consistently achieves state-of-the-art performance in both absolute and comparative evaluation settings.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of UrgentMOS, a unified framework for speech quality assessment that effectively leverages diverse metrics and pairwise preferences to improve robustness and generalization in evaluation scenarios. This work represents a meaningful step forward in the field, addressing critical challenges in speech quality assessment and providing a foundation for future research and applications.
Methodology
The proposed UrgentMOS framework introduces a novel architecture that integrates two main components: the Absolute Metric Prediction Module (AMPM) and the Naturalness-Conditioned Preference Module (NCPM). This dual approach allows for the simultaneous prediction of multiple quality metrics and the modeling of pairwise preferences, which is a significant advancement over traditional methods that often focus on single metrics. The methodology effectively addresses the challenge of training with heterogeneous datasets and partially annotated data, showcasing a robust learning paradigm that tolerates missing metrics during training. The use of cross-attention in the NCPM to model preferences is particularly innovative and indicates a thoughtful approach to leveraging existing data structures.
Results & evaluation
The experiments conducted are extensive, covering various speech quality datasets, including those with simulated distortions and real-world applications like speech enhancement and synthesis. The results demonstrate that UrgentMOS achieves state-of-the-art performance in both absolute and comparative evaluations, which is a strong indicator of its effectiveness. The thoroughness of the experimental setup, including diverse datasets, enhances the credibility of the findings and showcases the framework's versatility.
Reproducibility
While the paper outlines the architecture and methodology, it lacks detailed implementation specifics that would facilitate reproducibility. There is no mention of code availability or supplementary materials that could aid other researchers in replicating the study. This is a significant aspect that needs improvement, as reproducibility is crucial in machine learning research.
Limitations
The paper acknowledges that UrgentMOS does not provide explicit natural language explanations for its quality judgments, which could limit its applicability in contexts where interpretability is essential. Additionally, while the framework improves robustness with multiple feature extractors, it also increases inference costs, which may hinder its deployment in latency-sensitive applications. The potential degradation of performance with a large number of heterogeneous metrics is another limitation that the authors recognize, suggesting a need for future work to optimize metric selection.
Broader impact
The advancements presented in UrgentMOS have the potential to significantly impact the field of speech quality assessment, particularly in applications where human evaluation is impractical. By improving the robustness and generalization of speech quality models, this research could enhance the development of speech generation systems, leading to better user experiences in various domains, including telecommunications, virtual assistants, and entertainment. The main contribution of this paper is the introduction of UrgentMOS, a unified framework for speech quality assessment that effectively leverages diverse metrics and pairwise preferences to improve robustness and generalization in evaluation scenarios. This work represents a meaningful step forward in the field, addressing critical challenges in speech quality assessment and providing a foundation for future research and applications.
Author of this paper? Send a correction.