UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022
Abstract
We present the UTokyo-SaruLab mean opinion score (MOS) prediction system submitted to VoiceMOS Challenge 2022. The challenge is to predict the MOS values of speech samples collected from previous Blizzard Challenges and Voice Conversion Challenges for two tracks: a main track for in-domain prediction and an out-of-domain (OOD) track for which there is less labeled data from different listening tests. Our system is based on ensemble learning of strong and weak learners. Strong learners incorporate several improvements to the previous fine-tuning models of self-supervised learning (SSL) models, while weak learners use basic machine-learning methods to predict scores from SSL features. In the Challenge, our system had the highest score on several metrics for both the main and OOD tracks. In addition, we conducted ablation studies to investigate the effectiveness of our proposed methods.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the development of the UTMOS system for predicting mean opinion scores in speech synthesis, which combines advanced machine learning techniques to achieve high performance in a competitive challenge setting. This work is significant as it addresses the challenges of subjective evaluation in speech synthesis, providing a pathway for more efficient and scalable quality assessment methods.
Methodology
The methodology presented in this paper is robust, utilizing an ensemble learning approach that combines strong learners (fine-tuned self-supervised learning models) and weak learners (basic machine learning models). The introduction of contrastive learning and listener-dependent predictions is innovative, enhancing the model's ability to generalize across different datasets. The use of phoneme encoding and data augmentation techniques further strengthens the approach, making it suitable for both in-domain and out-of-domain predictions. The paper also includes ablation studies that provide valuable insights into the effectiveness of various components of the model.
Results & evaluation
The experimental evaluation is thorough, with results from the VoiceMOS Challenge 2022 demonstrating the system's effectiveness. The paper reports high performance metrics, including MSE and SRCC, across both the main and out-of-domain tracks. The inclusion of detailed experimental conditions and configurations enhances the credibility of the results. The use of multiple metrics for evaluation provides a comprehensive view of the model's performance.
Reproducibility
The paper provides a link to the implementation on GitHub, which is a positive aspect for reproducibility. However, the paper could benefit from more detailed descriptions of the experimental setup, including hyperparameter settings and data preprocessing steps, to facilitate easier replication by other researchers.
Limitations
One limitation is the reliance on synthetic datasets, which may not fully capture the complexities of real-world speech samples. Additionally, while the model performs well on the challenge datasets, its generalizability to other domains or languages remains to be tested. The paper also does not discuss potential biases in the listener ratings that could affect the MOS predictions.
Broader impact
The research has significant implications for the field of speech synthesis and quality assessment, particularly in developing automated systems for evaluating synthetic speech. The methods proposed could be applied to various applications, including voice assistants, automated customer service, and any domain where speech quality is critical. The advancements in MOS prediction could lead to improved user experiences in these applications. The main contribution of this paper is the development of the UTMOS system for predicting mean opinion scores in speech synthesis, which combines advanced machine learning techniques to achieve high performance in a competitive challenge setting. This work is significant as it addresses the challenges of subjective evaluation in speech synthesis, providing a pathway for more efficient and scalable quality assessment methods.
Curator note
Saeki et al.; MOS prediction model; standard automatic MOS estimator for TTS evaluation
Author of this paper? Send a correction.