FlowTTS-GRPO: Online Reinforcement Learning with Multi-Objective Reward Optimization for Flow-Matching Based Text-to-Speech
Abstract
Existing Reinforcement Learning (RL) research for Text-to-Speech (TTS) focuses on large language models (LLMs), leaving Flow-Matching (FM) under-explored. We present FlowTTS-GRPO, an online RL framework for FM-based TTS. By converting ordinary differential equation (ODE) trajectories into stochastic differential equation (SDE) paths, our method enables direct fine-tuning of open-source FM models without auxiliary models. We show that a weighted reward combination converges faster than a probabilistic scheme, and identify three practical optimizations: omitting classifier-free guidance (CFG) during training accelerates convergence; synthesizing hard cases improves robustness; and applying RL to the FM component enhances audio-detail metrics. Experiments on CosyVoice 3.0 and F5-TTS demonstrate objective and subjective preference gains in speaker similarity and perceptual quality, with F5-TTS also improving intelligibility.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper successfully adapts Group Relative Policy Optimization (GRPO) to Flow-Matching based TTS by introducing stochasticity via SDE conversion, demonstrating that online RL can significantly enhance speaker similarity and perceptual quality in both hybrid and pure FM architectures without auxiliary models.
Methodology
The paper proposes FlowTTS-GRPO, an online reinforcement learning framework tailored for Flow-Matching (FM) based Text-to-Speech (TTS) models. The core technical novelty lies in adapting the Group Relative Policy Optimization (GRPO) algorithm, originally designed for Large Language Models (LLMs), to continuous diffusion/flow models. This is achieved by converting the deterministic Ordinary Differential Equation (ODE) sampling trajectory into a Stochastic Differential Equation (SDE) path, thereby introducing the necessary stochasticity for policy gradient estimation. The authors formulate the FM decoding process as a Markov Decision Process (MDP) where actions are velocity predictions. They employ a multi-objective reward structure combining speaker similarity (SS), ASR-based intelligibility (CER/WER), and perceptual quality (DNSMOS). A key methodological contribution is the analysis of reward fusion strategies, demonstrating that weighted combination with standard deviation normalization converges faster and more stably than probabilistic assignment. Additionally, they identify practical optimizations such as omitting Classifier-Free Guidance (CFG) during training to enhance exploration and using hard-case synthesis to improve robustness. The approach is model-agnostic regarding the FM backbone, successfully applied to both LLM-FM hybrid (CosyVoice 3.0) and pure FM (F5-TTS) architectures.
Results & evaluation
The experimental evaluation is comprehensive, covering two major open-source TTS systems (CosyVoice 3.0 and F5-TTS) and multiple languages (Chinese, English, and several European languages). The authors utilize the Seed-TTS-Eval benchmark, reporting improvements in speaker similarity (surpassing closed-source baselines like Seed-TTS on SS1 for Chinese), perceptual quality (DNSMOS), and intelligibility (WER/CER). The inclusion of subjective A/B preference tests strengthens the claims by correlating objective metrics with human judgment. Ablation studies effectively isolate the impact of reward combination strategies, CFG omission, and hard-case training. The results demonstrate that RL on the FM component primarily enhances acoustic details and timbre, while RL on the LLM component (as seen in comparative baselines) is more critical for semantic alignment, providing valuable architectural insights.
Reproducibility
The paper provides sufficient detail for reproduction, including the MDP formulation, SDE conversion equations, reward definitions, and training hyperparameters (LoRA ranks, noise levels, window sizes). The use of widely available models (CosyVoice 3.0, F5-TTS, Whisper, Paraformer) and datasets (WenetSpeech4TTS, LibriTTS) facilitates replication. However, the specific implementation details of the SDE windowing and the exact weighting coefficients for the multi-objective reward are provided, though the code is not publicly linked in the text. The distinction between training and inference CFG usage is clearly explained.
Limitations
The primary limitation is the computational cost associated with online RL, requiring multiple rollouts per prompt and significant GPU resources (8 GPUs mentioned). The method relies on proxy rewards (DNSMOS, ASR, SS embeddings) which may not perfectly align with all aspects of human perception, although subjective tests mitigate this concern. The improvement in intelligibility for the LLM-FM hybrid (CosyVoice) is limited because the semantic content is determined by the frozen LLM front-end; RL on the FM can only refine acoustic realization, not correct semantic errors. Furthermore, the "hard case" synthesis strategy, while effective, relies on heuristic augmentations that may not cover all edge cases in natural speech.
Broader impact
This work significantly advances the field of generative audio by bridging the gap between discrete token-based RL (used in LLMs) and continuous flow-based generation. It enables the fine-tuning of high-quality, open-source FM models without the need for complex auxiliary reward models or value networks, democratizing access to advanced RL techniques in TTS. The findings on reward conflicts and optimization strategies provide generalizable insights for other continuous generative tasks. The potential for improved voice cloning and natural speech synthesis has broad applications in accessibility, entertainment, and human-computer interaction, though it also raises concerns regarding voice impersonation and deepfakes. The paper successfully adapts Group Relative Policy Optimization (GRPO) to Flow-Matching based TTS by introducing stochasticity via SDE conversion, demonstrating that online RL can significantly enhance speaker similarity and perceptual quality in both hybrid and pure FM architectures without auxiliary models.
Author of this paper? Send a correction.