GROW: Group-Relative Advantage-Weighted On-Policy Reinforcement Learning of Autoregressive-Diffusion Text-to-Speech model
Abstract
Reinforcement learning for flow-matching text-to-speech is complicated by deterministic ODE sampling: trajectory-level policy-gradient methods typically convert the ODE into an SDE and track per-step likelihood ratios, introducing stochastic perturbations and substantial overhead. We propose GROW, a group-relative advantage-weighted on-policy RL method that acts directly on the standard flow-matching objective. For each prompt, GROW samples a group of on-policy utterances, separately standardizes intelligibility and speaker-similarity rewards within the group, and combines them to reweight flow-matching regression. A Wasserstein-2 velocity penalty anchors the updated model to a frozen pretrained reference. A group-mean reward baseline is introduced to convert reward weighting into advantage weighting. For strong pretrained TTS models with concentrated rewards, positive exponential weighting is dominated by reward-agnostic self-imitation, whereas a zero-mean signed advantage preserves effective within-group credit assignment. Instantiated on DiTAR and evaluated on LibriSpeech and Seed-TTS EN/ZH, GROW reduces average WER from 2.016 to 1.558 and raises speaker similarity from 0.676 to 0.715 while keeping UTMOS. With 10-NFE training rollouts and 32-NFE evaluation, GROW retains comparable performance while training 2.9x faster than 32-NFE DiTAR-GRPO. We will open-source complete GROW codes, faithful DiTAR reproduction, and all model checkpoints.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
GROW introduces a novel, efficient, and theoretically motivated advantage-weighted regression framework for flow-matching TTS, demonstrating superior performance and training efficiency over trajectory-based RL methods. The paper makes a significant technical contribution by bridging the gap between discrete-token RL alignment techniques and continuous flow-matching generators, offering a scalable solution for post-training autoregressive-diffusion models.
Methodology
The paper proposes GROW, a reinforcement learning method tailored for autoregressive-diffusion Text-to-Speech (TTS) models, specifically instantiated on DiTAR. The core innovation lies in addressing the inefficiency and instability of converting deterministic ODE sampling (used in flow-matching) into Stochastic Differential Equations (SDEs) for policy gradient methods. Instead, GROW applies advantage-weighted regression directly to the flow-matching objective. It samples a group of on-policy rollouts, computes group-relative advantages for intelligibility and speaker similarity rewards, and uses these signed, normalized advantages to weight the standard flow-matching loss. A Wasserstein-2 velocity penalty is used to anchor the model to the pretrained reference, preventing divergence. The methodology is theoretically grounded, drawing on Taylor expansions to justify why linear advantage weighting outperforms exponential weighting in the high-reward, concentrated regime typical of strong pretrained TTS models. The approach is elegant in its simplicity, avoiding the overhead of trajectory storage and per-step likelihood ratio calculations required by methods like GRPO or PPO in continuous spaces.
Results & evaluation
The experimental evaluation is comprehensive and robust. The authors compare GROW against a strong baseline (DiTAR-GRPO) and the pretrained model across three benchmarks: LibriSpeech-PC, Seed-TTS EN, and Seed-TTS ZH. Results show consistent improvements in Word Error Rate (WER) and Speaker Similarity (SIM) while maintaining or slightly improving UTMOS. Notably, GROW achieves these gains with significantly higher training efficiency, being 2.9x faster than the 32-NFE DiTAR-GRPO baseline when using 10-NFE rollouts. The ablation studies are particularly strong, providing detailed analysis on the impact of rollout NFE, CFG settings, learning rates, W2 anchor strength, and reward design. The decomposition of self-imitation vs. reward-contrast provides valuable insight into why the proposed weighting scheme works better than exponential alternatives.
Reproducibility
The paper provides extensive implementation details, including model architecture (Qwen3-0.6B backbone, LocDiT), hyperparameters (learning rates, group size, NFEs), and dataset descriptions. The authors explicitly state they will open-source the code, reproduction, and checkpoints, which significantly enhances reproducibility. The algorithm is clearly defined with pseudocode. The use of standard metrics (WER, SIM, UTMOS) and public datasets further supports reproducibility.
Limitations
The method is currently evaluated primarily on zero-shot TTS tasks using specific reward models (ASR for WER, Speaker Encoder for SIM). The generalizability to other TTS objectives (e.g., emotion, prosody) or other flow-matching architectures (e.g., F5-TTS) is suggested but not extensively demonstrated. The reliance on frozen reward models introduces potential biases inherent in those models (e.g., ASR errors affecting WER rewards). Additionally, the performance gain, while consistent, is modest in absolute terms (e.g., WER reduction of ~0.45), though significant in the context of already high-performing models. The method's effectiveness on very short or very long utterances is not explicitly analyzed.
Broader impact
This work contributes to the efficient alignment of generative audio models, potentially lowering the computational cost of post-training for high-quality TTS systems. This could democratize access to advanced speech synthesis technologies. However, as with any TTS advancement, there are risks associated with improved voice cloning capabilities, including potential misuse for deepfakes or misinformation. The authors' open-source commitment promotes transparency and further research in this area. GROW introduces a novel, efficient, and theoretically motivated advantage-weighted regression framework for flow-matching TTS, demonstrating superior performance and training efficiency over trajectory-based RL methods. The paper makes a significant technical contribution by bridging the gap between discrete-token RL alignment techniques and continuous flow-matching generators, offering a scalable solution for post-training autoregressive-diffusion models.
Author of this paper? Send a correction.