Multi-Reward GRPO for Stable and Prosodic Single-Codebook TTS LLMs at Scale
Abstract
Recent advances in Large Language Models (LLMs) have transformed text-to-speech (TTS) synthesis, inspiring autoregressive frameworks that represent speech as sequences of discrete codec tokens. Among them, single-codebook TTS LLMs have emerged as compact and streamable architectures that jointly model semantic and acoustic integration. However, despite their efficiency, these models often exhibit unstable prosody, speaker drift, and degraded naturalness. To address these issues, we propose a multi-reward Group Relative Policy Optimization (GRPO) framework that directly optimizes the token generation policy of single-codebook TTS LLMs. Beyond standard intelligibility and speaker similarity objectives, our design integrates three rule-based rewards: a length penalty for duration consistency, an entropy regularization reward for decoding stability, and an LLM-annotated prosody alignment reward that explicitly supervises rhythm. In this prosody reward, an external reasoning LLM predicts multiple plausible pause structures via in-context learning, providing a human-preference-aligned supervisory signal for GRPO training. To assess universality, we further attach a flow-matching (FM) decoder on top of the GRPO-optimized AR backbone and observe consistent additional gains, indicating that our reinforcement optimization enhances the intrinsic AR policy. We further conduct a scalability analysis across data sizes and model scales, revealing that the proposed method consistently enhances prosodic stability, speaker similarity, and overall speech naturalness in single-codebook TTS LLMs.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel multi-reward GRPO framework that significantly enhances the performance of single-codebook TTS LLMs by addressing key challenges in prosody and speaker similarity. The comprehensive methodology and rigorous experimental evaluation contribute valuable insights to the field of TTS synthesis, with the potential for broad applications in human-computer interaction.
Methodology
The paper introduces a multi-reward Group Relative Policy Optimization (GRPO) framework that enhances the token generation policy of single-codebook TTS LLMs. The integration of multiple rule-based rewards (length penalty, entropy regularization, and prosody alignment) is a novel approach that addresses common issues in TTS systems, such as prosody instability and speaker drift. The use of an external reasoning LLM to predict pause structures for prosody alignment is particularly innovative, leveraging in-context learning to provide a human-preference-aligned supervisory signal. The methodology is well-structured, with clear definitions of the reward functions and their intended impacts on the model's performance.
Results & evaluation
The experiments are comprehensive, utilizing a large bilingual corpus and various evaluation metrics (CER, SIM, MOS) to assess the effectiveness of the proposed framework. The results demonstrate significant improvements in prosodic stability, speaker similarity, and naturalness compared to existing models. The scalability analysis across different model sizes and data scales adds depth to the evaluation, showing that the proposed method is effective across a range of conditions. The ablation study further validates the contribution of each reward component, providing insights into their individual impacts on performance.
Reproducibility
The paper provides detailed implementation details, including the architecture, training configurations, and data sources. However, the absence of a public code repository or demo URL limits the reproducibility of the results. While the methodology is well-explained, the lack of accessible resources may hinder other researchers from replicating the study.
Limitations
One limitation of the study is the reliance on a specific reasoning LLM for prosody alignment, which may not generalize across all languages or dialects. Additionally, while the results are promising, the paper does not address potential computational costs associated with the proposed GRPO framework, particularly in terms of training time and resource requirements. The evaluation is primarily focused on objective metrics, and further subjective assessments could strengthen the findings.
Broader impact
The proposed framework has significant implications for the field of TTS synthesis, particularly in enhancing the naturalness and expressivity of synthesized speech. Improved prosody and speaker similarity can lead to more engaging and human-like interactions in applications such as virtual assistants, audiobooks, and language learning tools. The integration of reinforcement learning in TTS systems could pave the way for more adaptive and context-aware speech synthesis technologies. The paper presents a novel multi-reward GRPO framework that significantly enhances the performance of single-codebook TTS LLMs by addressing key challenges in prosody and speaker similarity. The comprehensive methodology and rigorous experimental evaluation contribute valuable insights to the field of TTS synthesis, with the potential for broad applications in human-computer interaction.
Author of this paper? Send a correction.