VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation
Abstract
Recently, zero-shot text-to-speech (TTS) has enabled high-fidelity and expressive speech synthesis, but it often fails to imitate unseen speaking styles from uncommon scenarios (e.g., crosstalk, dialects). Moreover, fine-tuning pretrained models requires large, high-quality datasets, limiting rapid personalization. We propose VoiceTTA, a reinforcement learning-based test-time adaptation (TTA) method that improves voice imitation of pretrained zero-shot TTS models. VoiceTTA introduces two style rewards based on coefficient-of-variation differences of F0 and energy, combined with speaker similarity and intelligibility (WER from a pretrained Whisper model), and optimizes learnable prefixes via group relative preference optimization (GRPO) in a flow matching-based model at inference time. Extensive experiments demonstrate substantial improvements on uncommon speech prompts, outperforming state-of-the-art baselines. Audio samples are available at https://voicetta.pages.dev/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces VoiceTTA, a reinforcement learning-based test-time adaptation method that optimizes learnable prefixes in flow-matching TTS models using a composite reward of prosodic variation and speaker similarity. The work represents a novel application of LLM-centric RL algorithms (GRPO) to continuous audio generation, offering a parameter-efficient way to adapt zero-shot TTS models to unseen, low-resource styles. While the technical approach is innovative and the results show clear improvements in objective style similarity metrics, the reliance on an internal dataset and the modest gains in perceptual naturalness limit its immediate impact on the broader TTS community. It serves as a proof-of-concept for RL-based TTA in audio, paving the way for more sophisticated reward designs and public benchmarking.
Methodology
The paper proposes VoiceTTA, a test-time adaptation (TTA) framework for zero-shot Text-to-Speech (TTS) models. The core innovation lies in applying Group Relative Policy Optimization (GRPO)—an algorithm typically associated with Large Language Model (LLM) alignment—to optimize learnable prefixes in a flow-matching-based TTS model during inference. The method introduces a composite reward function consisting of style rewards (Coefficient of Variation differences for F0 and energy) and a speaker similarity reward, balanced with an intelligibility reward (Word Error Rate from Whisper). The approach is technically sound in its adaptation of RL techniques to continuous generation tasks, although the use of CV differences as a proxy for prosodic style is a simplification that may not capture complex temporal dynamics. The derivation of the probability ratio using flow-matching loss as a proxy is a clever workaround for the lack of discrete token probabilities in diffusion/flow models.
Results & evaluation
The experiments are conducted on a custom internal dataset of "uncommon" speech styles (accented, children, slurred, dialects) and the KeSpeech dialect dataset. The baseline comparisons include F5-TTS, CosyVoice, MaskGCT, and Vevo. The results show improvements in Speaker Similarity (S-SIM) and Word Error Rate (WER) compared to the base F5-TTS model. However, the subjective evaluation (MOS) shows only marginal improvements in style similarity (3.27 vs 3.07 for F5-TTS) and a slight drop in naturalness compared to CosyVoice. The use of an internal, undisclosed dataset for the primary "uncommon" evaluation is a significant limitation for reproducibility and fair comparison. The ablation studies provide some insight into the reward weights and number of prefixes, but the overall performance gains, while statistically significant in objective metrics, appear modest in perceptual quality.
Reproducibility
The paper provides hyperparameters for the GRPO optimization (learning rate, number of prefixes, candidate sampling temperature range). However, the primary evaluation dataset is internal and not publicly available, which severely hinders reproducibility. The code is not explicitly linked in the text (only a demo page is provided), and the specific versions of the backbone models (F5-TTS, Whisper, speaker embedding models) are not fully detailed. The reliance on a "pretrained Whisper model" for WER calculation is standard, but the exact configuration is needed for exact replication.
Limitations
The primary limitation is the lack of public data for the main experimental claims. The use of Coefficient of Variation for F0 and Energy is a coarse metric for prosody and may fail to capture nuanced stylistic elements like rhythm or phrasing. The GRPO adaptation is performed at inference time, which adds computational overhead per utterance, potentially limiting real-time applicability despite the "lightweight" parameter claim. The subjective MOS scores are low across the board (around 3.0-3.5), suggesting that while the method improves similarity metrics, the overall audio quality remains mediocre compared to state-of-the-art systems trained on massive datasets.
Broader impact
This work contributes to the field of efficient model adaptation, demonstrating that RL-based TTA can be effective for audio generation tasks. It highlights the potential for personalizing large generative models without full fine-tuning. However, the reliance on proprietary internal data limits the broader scientific impact. The method could be valuable for niche applications where data collection is difficult, but the marginal gains in naturalness may limit widespread adoption over existing fine-tuned or larger zero-shot models. This paper introduces VoiceTTA, a reinforcement learning-based test-time adaptation method that optimizes learnable prefixes in flow-matching TTS models using a composite reward of prosodic variation and speaker similarity. The work represents a novel application of LLM-centric RL algorithms (GRPO) to continuous audio generation, offering a parameter-efficient way to adapt zero-shot TTS models to unseen, low-resource styles. While the technical approach is innovative and the results show clear improvements in objective style similarity metrics, the reliance on an internal dataset and the modest gains in perceptual naturalness limit its immediate impact on the broader TTS community. It serves as a proof-of-concept for RL-based TTA in audio, paving the way for more sophisticated reward designs and public benchmarking.
Author of this paper? Send a correction.