RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTS
Abstract
Differentiable reinforcement learning (RL) frameworks like DiffRO offer a powerful approach for controllable text-to-speech (TTS), but are vulnerable to reward hacking, particularly for nuanced tasks like emotion control. The policy model can exploit a vanilla Reward Model (RM) by generating acoustic artifacts to achieve spurious rewards, but at the cost of degrading perceptual quality. To address this, we propose Robust Reward Policy Optimization (RRPO), a novel framework that employs a hybrid regularization scheme. This scheme develops a robust RM whose reward signal is more reliably aligned with human perception, compelling the policy to abandon detrimental shortcuts and instead learn the complex features of genuine emotions. Our ablation study confirms the enhanced robustness of our RM, as evidenced by its strong cross-lingual generalization. The subjective evaluation demonstrates that this robust RM effectively mitigates reward hacking, leading to significant improvements in both emotional expressiveness and naturalness over all baselines. Demo page: https://lrwinr.github.io/RRPO-CosyVoice.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of the RRPO framework, which effectively mitigates reward hacking in emotional TTS systems through a novel hybrid regularization scheme. This work represents a significant advancement in the field, addressing critical challenges in generating emotionally expressive and natural speech synthesis while providing a robust methodology that can be adapted to other applications.
Methodology
The proposed Robust Reward Policy Optimization (RRPO) framework introduces a hybrid regularization scheme that effectively addresses the vulnerabilities of existing differentiable reinforcement learning methods in emotional text-to-speech (TTS). The methodology is well-structured, leveraging three distinct regularization techniques—Label Smoothing, Energy-Adaptive Mixup, and Adversarial Training—to enhance the robustness of the reward model (RM). Each component is justified with clear explanations of how they mitigate specific issues related to reward hacking, making the approach both innovative and practical. However, the complexity of the hybrid scheme may pose challenges in terms of implementation and tuning.
Results & evaluation
The experiments are comprehensive, involving both subjective and objective evaluations to demonstrate the effectiveness of RRPO. The use of a high-quality emotional dataset and a well-defined evaluation methodology, including Mean Opinion Score (MOS) assessments, strengthens the validity of the results. The ablation study provides insights into the contributions of each regularization component, although further exploration of the trade-offs between generalization and adversarial robustness could enhance understanding.
Reproducibility
The paper provides sufficient implementation details, including the architecture of the RM and the training setup. However, the lack of a publicly available code repository limits the reproducibility of the results. Providing access to the trained models and code would significantly enhance the ability of other researchers to validate and build upon this work.
Limitations
One limitation is the reliance on a single language (Mandarin) for training and evaluation, which may affect the generalizability of the findings to other languages or dialects. Additionally, while the hybrid regularization scheme shows promise, its complexity may hinder practical application in real-world scenarios. The potential for overfitting to the small, high-quality dataset used for fine-tuning the RM is also a concern.
Broader impact
The proposed RRPO framework has the potential to significantly improve emotional TTS systems, enhancing human-computer interaction through more expressive and natural speech synthesis. The robustness of the RM against reward hacking can lead to more reliable applications in various domains, including virtual assistants, gaming, and mental health support. The adaptability of the hybrid regularization scheme for other acoustic tasks suggests broader implications for the field of audio processing and machine learning. The main contribution of this paper is the introduction of the RRPO framework, which effectively mitigates reward hacking in emotional TTS systems through a novel hybrid regularization scheme. This work represents a significant advancement in the field, addressing critical challenges in generating emotionally expressive and natural speech synthesis while providing a robust methodology that can be adapted to other applications.
Author of this paper? Send a correction.