Training a Compact, Strong, and Realistic Text-to-Speech Model
Abstract
Text-to-speech (TTS) systems are conventionally optimized for intelligibility and speaker similarity. However, neither objective captures the prosodic detail that distinguishes human performances from synthetic speech, leaving a realism gap that larger models and higher-quality data can narrow but not close. In the absence of an objective that directly targets this gap, realism remains a persistent challenge in TTS. We present Audio8-TTS, a compact 0.6B-parameter model for zero-shot voice cloning that achieves state-of-the-art performance in intelligibility and speaker similarity, with particularly strong results in perceptual naturalness. To align the training signal with human perception rather than signal-level fidelity, we construct a quality ladder from widely used open-source TTS systems and train a realism critic to assess generated speech against matched human recordings of the same sentences. An identity-erasure objective discourages the critic from relying on generator fingerprints. We train the model in three stages: multilingual supervised fine-tuning; group relative policy optimization (GRPO) with word error rate (WER) and speaker similarity (SIM) rewards; and GRPO guided by the realism critic. The final stage improves perceived prosodic naturalness and speaker similarity while preserving WER performance.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents Audio8-TTS, a compact 0.6B parameter zero-shot voice cloning model that achieves state-of-the-art perceptual naturalness by employing a reference-based realism critic trained on a quality ladder of open-source TTS systems, integrated into a Group Relative Policy Optimization (GRPO) framework to optimize prosodic realism alongside intelligibility and speaker similarity.
Methodology
The paper proposes a three-stage training framework for a compact (0.6B) dual-autoregressive TTS model. The core novelty lies in the third stage: using a learned "realism critic" as a reward model for Group Relative Policy Optimization (GRPO). The critic is trained on a "quality ladder" constructed from outputs of various open-source TTS systems, ranked via a private blind listening study, with human recordings as the top anchor. A key technical contribution is the "identity-erasure" objective, which uses pause-compression degradation to prevent the critic from relying on generator-specific fingerprints rather than prosodic quality. The architecture itself (fast-slow dual autoregressive) is standard in recent codec-LM literature, but the specific application of reference-based realism rewards to compact models is a distinct methodological contribution.
Results & evaluation
The evaluation is comprehensive, covering 9 CV3 languages and 13 additional languages. The model achieves competitive WER and SIM scores compared to much larger models (e.g., Seed-TTS, MiniMax). The most significant evidence is the blind human evaluation, where the realism-optimized model received a substantially higher Elo rating than the SFT and WER/SIM-only models, validating the effectiveness of the realism critic. The ablation of the critic's generalization to unseen systems (Fish Audio S2, Spark-TTS) is a strong point, showing the critic learns quality rather than memorizing specific generators.
Reproducibility
The authors release code and model weights on GitHub. They provide detailed descriptions of the data mixture, training stages, and critic construction. However, the "private blind listening study" and "private collection" of data limit full reproducibility of the critic's training data. The use of specific open-source systems for the quality ladder is reproducible, but the exact ranking and degradation parameters are proprietary.
Limitations
The realism critic requires a matched human reference for every sentence during inference/evaluation, which is a significant practical constraint for zero-shot cloning where such references may not exist. The paper acknowledges that the critic's signal is primarily driven by pause structure due to the degradation method used, potentially neglecting other prosodic cues like pitch and energy. Additionally, the model struggles with low-resource languages (e.g., Thai, Finnish) where error rates remain high.
Broader impact
This work demonstrates that compact TTS models can achieve high perceptual naturalness through targeted preference optimization, not just scale. The approach of using a learned critic to bridge the gap between objective metrics (WER/SIM) and subjective realism is applicable to other generative audio tasks. The identity-erasure technique is a valuable contribution to the field of reward model design for generative models. The paper presents Audio8-TTS, a compact 0.6B parameter zero-shot voice cloning model that achieves state-of-the-art perceptual naturalness by employing a reference-based realism critic trained on a quality ladder of open-source TTS systems, integrated into a Group Relative Policy Optimization (GRPO) framework to optimize prosodic realism alongside intelligibility and speaker similarity.
Author of this paper? Send a correction.